Computer Science

Computer Science ›› 2022, Vol. 49 ›› Issue (7): 236-241.doi: 10.11896/jsjkx.211200071

• Computer Network • Previous Articles     Next Articles

Analysis of Performance Metrics of Semantic Communication Systems

JIANG Sheng-teng, ZHANG Yi-chi, LUO Peng, LIU Yue-ling, CAO Kuo, ZHAO Hai-tao, WEI Ji-bo   

  1. College of Electronic Science and Technology,National University of Defense Technology,Changsha 410073,China
  • Received:2021-12-06 Revised:2022-04-04 Online:2022-07-15 Published:2022-07-12
  • About author:JIANG Sheng-teng,born in 1998,postgraduate.His main research interests include semantic communication and so on.
    CAO Kuo,born in 1990,Ph.D,lecturer.His main research interests include cooperative communications and physical layer security of wireless communications.
  • Supported by:
    National Natural Science Foundation of China(61931020,U19B2024,62001483).

PDF (PC)

703

Abstract: Semantic communication system is currently a hot research topic in the communication field,but a mature evaluation system has not been established in this field,which leads to the different performance of semantic communication systems designed under different performance metrics.This paper mainly focuses on semantic communication systems,introducing perfor-mance metrics based on precision,performance metrics based on recall,performance metrics based on the combination of precision and recall,and performance metrics based on word vector space models.It also elaborates on the background,significance,main algorithm ideas,and scope of application of various performance metrics in semantic communication,and analyzes and compares the differences,advantages and disadvantages of each performance metric.Finally,it summarizes the problems faced by semantic communication performance metrics at this stage,and points out the future development direction of performance metrics research in semantic communication system.

Key words: Performance measurement, Semantic communication, Text evaluation, Word vector

CLC Number: 

  • TN919
[1]SHI G M,LI Y Y,XIE X M.Semantic communications:outcome of the intelligence era[J].Pattern Recognition and Artificial Intelligence,2018,30(3):289-299.
[2]SHANNON C.A mathematical theory of communication[J].Bell System Technical Journal,1948,27(3):379-423.
[3]BAO J,BASU P,DEAN M,et al.Towards a theory of semantic communication[C]//2011 IEEE Network Science Workshop.IEEE,2011:110-117.
[4]GÜLER B,YENER A,SWAMI A.The semantic communication game[J].IEEE Transactions on Cognitive Communications and Networking,2018,4(4):787-802.
[5]WENG Z Z,QIN Z J,LI G.Semantic communications for speech signals[C]//ICC 2021 - IEEE International Conference on Communications.IEEE,2021:1-6.
[6]JUBA B,SUDAN M.Universal semantic communication I[C]//Proceedings of the Fortieth Annual ACM Symposium on Theory of Computing.2008:123-132.
[7]PAPINENI K,ROUKOS S,WARD T,et al.BLEU:a method for automatic evaluation of machine translation[C]//40th Annual Meeting of the Association for Computational Linguistics.2002:311-318.
[8]DODDINGTON G.Automatic evaluation of machine translation quality using n-gram cooccurrence statistics[C]//Proceedings of the 2nd International Conference on Human Language Techno-logy Research.2002:138-145.
[9]LIN C Y.Rouge:a package for automatic evaluation of summaries[C]//Proceedings of the Workshop on Text Summarization Branches Out.2004:74-81.
[10]BANERJEE S,LAVIE A.Meteor:an automatic metric for MT evaluation with improved correlation with human judgments[C]//Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.2005:65-72.
[11]AGIRRE E,BANEA C,CER D,et al.SemEval-2016 Task 1:semantic textual similarity,monolingual and cross-lingual evalu-ation[C]//Proceedings of SemEval-2016.2016:497-511.
[12]VEDANTAM R,ZITNICK C,PARIKH D.Cider:consensus-based image description evaluation[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2015:4566-4575.
[13]GOMAA W,FAHMY A.A survey of text similarity approaches[J].International Journal of Computer Applications,2013,68(13):13-17.
[14]HOVY E.Toward finely differentiated evaluation metrics formachine translation[C]//Proceedings of the EAGLES Workshop on Standards and Evaluation.1999:127-133.
[15]POPOVIC M.CHRF:character n-gram F-score for automatic MT evaluation[C]//Proceedings of the Tenth Workshop on Statistical Machine Translation.2015:392-395.
[16]MATHUR N,BALDWIN T,COHN T.Tangled up in BLEU:reevaluating the evaluation of automatic machine translation evaluation metrics[C]//Proceedings of the 58th Annual Mee-ting of the Association for Computational Linguistics.2020:4984-4997.
[17]REITER E.A structured review of the validity of BLEU[J].Computational Linguistics,2018,44(3):393-401.
[18]XIE H Q,QIN Z J.A lite distributed semantic communication system for Internet of Things[J].IEEE Journal on Selected Areas in Communications,2021,39(1):142-153.
[19]SMITH A,HARDMEIER C,TIEDEMANN J.Climbing Mont BLEU:the strange world of reachable High-BLEU translations[C]//Proceedings of the 19th Annual Conference of the Euro-pean Association for Machine Translation.2016:269-281.
[20]STENT A,MARGE M,SINGHAI M.Evaluating evaluationmethods for generation in the presence of variation[C]//Computational Linguistics and Intelligent Text Processing.2005:341-351.
[21]WHITE J,CONNELL T O,MARA F O.The ARPA MT evaluation methodologies:evolution,lessons,and future approaches[C]//Proceedings of the First Conference of the Association for Machine Translation in the Americas.1994:193-205.
[22]WISEMAN S,SHIEBER S,RUSH A.Challenges in data-to-document generation[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.2017:2253-2263.
[23]LAVIE A,SAGAE K,JAYARAMAN S.The significance of recall in automatic metrics for MT evaluation[C]//6th Conference of the Association for Machine Translation in the Americas.2004:134-143.
[24]ANDERSON P,FERNANDO B,JOHNSON M,et al.SPICE:semantic propositional image caption evaluation[C]//14th European Conference on Computer Vision.2016:382-398.
[25]BENGIO Y,DUCHARME R,VINCENT P,et al.A neuralprobabilistic language model[J].Journal of Machine Learning Research 3,2003,3(6):1137-1155.
[26]MAJUMDER G,PAKRAY P,GELBUKH A,et al.Semantictextual similarity methods,tools,and applications:a survey[J].Computaciony Sistemas,2016,20(4):647-665.
[27]MIKOLOV T,CHEN K,CORRADO G,et al.Efficient estimation of word representations in vector space[J].arXiv:1301.3781,2013.
[28]WANG Q,XU J G,CHEN H,et al.Two improved continuous bag-of-word models[C]//2017 International Joint Conference on Neural Networks.IEEE,2017:2851-2856.
[29]PENNINGTON J,SOCHER R,MANNING C.Glove:globalvectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Proces-sing.2014:1532-1543.
[30]DEVLIN J,CHANG M,LEE K,et al.Bert:pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics.2019:4171-4186.
[31]JIN X L,ZHANG S W,LIU J.Word semantic similarity calculation based on Word2vec[C]//2018 International Conference on Control,Automation and Information Sciences.IEEE,2018:12-16.
[32]HUANG P S,HE X D,GAO J F,et al.Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM International Conference on Information & Knowledge Management.2013:2333-2338.
[33]SHEN Y L,HE X D,GAO J F,et al.A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval[C]//Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management.2014:101-110.
[34]CER D,DIAB M,AGIRRE E,et al.SemEval-2017 Task 1:semantic textual similarity multilingual and crosslingual focused evaluation[C]//Proceedings of the 11th International Workshop on Semantic Evaluation.2017:1-14.
[35]WANG M,HU F H.The application of NLTK library for python natural language processing in corpus research[J].Theory and Practice in Language Studies,2021,11(9):1041-1049.
[36]BIRD S.NLTK:the natural language toolkit[C]//Proceedings of the COLING/ACL on Interactive Presentation Sessions.2006:69-72.
[1] LIU Shuo, WANG Geng-run, PENG Jian-hua, LI Ke. Chinese Short Text Classification Algorithm Based on Hybrid Features of Characters and Words [J]. Computer Science, 2022, 49(4): 282-287.
[2] LIN Li-xiang, LIU Xu-dong, LIU Shao-teng, XU Yue-dong. Survey on the Application of Forward Error Correction Coding in Network Transmission Protocols [J]. Computer Science, 2022, 49(2): 292-303.
[3] YANG Jin-cai, CAO Yuan, HU Quan, SHEN Xian-jun. Relation Classification of Chinese Causal Compound Sentences Based on Transformer Model and Relational Word Feature [J]. Computer Science, 2021, 48(6A): 295-298.
[4] YANG Qing, ZHANG Ya-wen, ZHU Li, WU Tao. Text Sentiment Analysis Based on Fusion of Attention Mechanism and BiGRU [J]. Computer Science, 2021, 48(11): 307-311.
[5] JING Li, LI Man-man, HE Ting-ting. Sentiment Classification of Network Reviews Combining Extended Dictionary and Self-supervised Learning [J]. Computer Science, 2020, 47(11A): 78-82.
[6] WANG Le-le,WANG Bin-qiang,LIU Jian-gang,ZHANG Jian-hui,MIAO Qi-guang. Study on Malicious Program Detection Based on Recurrent Neural Network [J]. Computer Science, 2019, 46(7): 86-90.
[7] LI Zhou-jun,WANG Chang-bao. Survey on Deep-learning-based Machine Reading Comprehension [J]. Computer Science, 2019, 46(7): 7-12.
[8] SHAN Yi-dong, WANG Heng-jun, HUANG He, YAN Qian. Study on Named Entity Recognition Model Based on Attention Mechanism——Taking Military Text as Example [J]. Computer Science, 2019, 46(6A): 111-114.
[9] ZHENG Cheng, HONG Tong-tong, XUE Man-yi. BLSTM_MLPCNN Model for Short Text Classification [J]. Computer Science, 2019, 46(6): 206-211.
[10] WU Chen, YUAN Yu-wei, WANG Hong-wei, LIU Yu, LIU Si-tong, QUAN Ji-cheng. Word Vectors Fusion Based Remote Sensing Scenes Zero-shot Classification Algorithm [J]. Computer Science, 2019, 46(12): 286-291.
[11] ZHENG Cheng, XUE Man-yi, HONG Tong-tong, SONG Fei-bao. DC-BiGRU_CNN Model for Short-text Classification [J]. Computer Science, 2019, 46(11): 186-192.
[12] WANG Wei, CAI Wan-Dong, LI Yong-Jun （School of Computer Science, Northwestern Polytechnical University, Xi＇an 710072）. [J]. Computer Science, 2007, 34(5): 45-47.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.