计算机科学 ›› 2022, Vol. 49 ›› Issue (7): 236-241.doi: 10.11896/jsjkx.211200071

• 计算机网络 • 上一篇    下一篇

语义通信系统的性能度量指标分析

姜胜腾, 张亦弛, 罗鹏, 刘月玲, 曹阔, 赵海涛, 魏急波   

  1. 国防科技大学电子科学学院 长沙410073
  • 收稿日期:2021-12-06 修回日期:2022-04-04 出版日期:2022-07-15 发布日期:2022-07-12
  • 通讯作者: 曹阔(caokuo90@sina.cn)
  • 作者简介:(jiangshengteng@nudt.edu.cn)
  • 基金资助:
    国家自然科学基金(61931020,U19B2024,62001483)

Analysis of Performance Metrics of Semantic Communication Systems

JIANG Sheng-teng, ZHANG Yi-chi, LUO Peng, LIU Yue-ling, CAO Kuo, ZHAO Hai-tao, WEI Ji-bo   

  1. College of Electronic Science and Technology,National University of Defense Technology,Changsha 410073,China
  • Received:2021-12-06 Revised:2022-04-04 Online:2022-07-15 Published:2022-07-12
  • About author:JIANG Sheng-teng,born in 1998,postgraduate.His main research interests include semantic communication and so on.
    CAO Kuo,born in 1990,Ph.D,lecturer.His main research interests include cooperative communications and physical layer security of wireless communications.
  • Supported by:
    National Natural Science Foundation of China(61931020,U19B2024,62001483).

摘要: 语义通信系统是目前通信领域的研究热点,但是该领域尚未建立起成熟的评价体系,导致不同性能度量指标下设计的语义通信系统的性能也各不相同。文中主要针对语义通信系统,介绍了基于精确率的性能度量指标、基于召回率的性能度量指标、基于精确率与召回率相结合的性能度量指标以及基于词向量空间模型的性能度量指标;并详细阐述了语义通信中各种性能度量指标提出的背景、意义、主要算法思想和适用范围,对比分析了每一种性能度量指标间的差异和优缺点。最后,总结了现阶段语义通信性能度量指标面临的问题,并展望了语义通信系统中性能度量指标研究的未来发展方向。

关键词: 词向量, 文本评价, 性能度量, 语义通信

Abstract: Semantic communication system is currently a hot research topic in the communication field,but a mature evaluation system has not been established in this field,which leads to the different performance of semantic communication systems designed under different performance metrics.This paper mainly focuses on semantic communication systems,introducing perfor-mance metrics based on precision,performance metrics based on recall,performance metrics based on the combination of precision and recall,and performance metrics based on word vector space models.It also elaborates on the background,significance,main algorithm ideas,and scope of application of various performance metrics in semantic communication,and analyzes and compares the differences,advantages and disadvantages of each performance metric.Finally,it summarizes the problems faced by semantic communication performance metrics at this stage,and points out the future development direction of performance metrics research in semantic communication system.

Key words: Performance measurement, Semantic communication, Text evaluation, Word vector

中图分类号: 

  • TN919
[1]SHI G M,LI Y Y,XIE X M.Semantic communications:outcome of the intelligence era[J].Pattern Recognition and Artificial Intelligence,2018,30(3):289-299.
[2]SHANNON C.A mathematical theory of communication[J].Bell System Technical Journal,1948,27(3):379-423.
[3]BAO J,BASU P,DEAN M,et al.Towards a theory of semantic communication[C]//2011 IEEE Network Science Workshop.IEEE,2011:110-117.
[4]GÜLER B,YENER A,SWAMI A.The semantic communication game[J].IEEE Transactions on Cognitive Communications and Networking,2018,4(4):787-802.
[5]WENG Z Z,QIN Z J,LI G.Semantic communications for speech signals[C]//ICC 2021 - IEEE International Conference on Communications.IEEE,2021:1-6.
[6]JUBA B,SUDAN M.Universal semantic communication I[C]//Proceedings of the Fortieth Annual ACM Symposium on Theory of Computing.2008:123-132.
[7]PAPINENI K,ROUKOS S,WARD T,et al.BLEU:a method for automatic evaluation of machine translation[C]//40th Annual Meeting of the Association for Computational Linguistics.2002:311-318.
[8]DODDINGTON G.Automatic evaluation of machine translation quality using n-gram cooccurrence statistics[C]//Proceedings of the 2nd International Conference on Human Language Techno-logy Research.2002:138-145.
[9]LIN C Y.Rouge:a package for automatic evaluation of summaries[C]//Proceedings of the Workshop on Text Summarization Branches Out.2004:74-81.
[10]BANERJEE S,LAVIE A.Meteor:an automatic metric for MT evaluation with improved correlation with human judgments[C]//Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.2005:65-72.
[11]AGIRRE E,BANEA C,CER D,et al.SemEval-2016 Task 1:semantic textual similarity,monolingual and cross-lingual evalu-ation[C]//Proceedings of SemEval-2016.2016:497-511.
[12]VEDANTAM R,ZITNICK C,PARIKH D.Cider:consensus-based image description evaluation[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2015:4566-4575.
[13]GOMAA W,FAHMY A.A survey of text similarity approaches[J].International Journal of Computer Applications,2013,68(13):13-17.
[14]HOVY E.Toward finely differentiated evaluation metrics formachine translation[C]//Proceedings of the EAGLES Workshop on Standards and Evaluation.1999:127-133.
[15]POPOVIC M.CHRF:character n-gram F-score for automatic MT evaluation[C]//Proceedings of the Tenth Workshop on Statistical Machine Translation.2015:392-395.
[16]MATHUR N,BALDWIN T,COHN T.Tangled up in BLEU:reevaluating the evaluation of automatic machine translation evaluation metrics[C]//Proceedings of the 58th Annual Mee-ting of the Association for Computational Linguistics.2020:4984-4997.
[17]REITER E.A structured review of the validity of BLEU[J].Computational Linguistics,2018,44(3):393-401.
[18]XIE H Q,QIN Z J.A lite distributed semantic communication system for Internet of Things[J].IEEE Journal on Selected Areas in Communications,2021,39(1):142-153.
[19]SMITH A,HARDMEIER C,TIEDEMANN J.Climbing Mont BLEU:the strange world of reachable High-BLEU translations[C]//Proceedings of the 19th Annual Conference of the Euro-pean Association for Machine Translation.2016:269-281.
[20]STENT A,MARGE M,SINGHAI M.Evaluating evaluationmethods for generation in the presence of variation[C]//Computational Linguistics and Intelligent Text Processing.2005:341-351.
[21]WHITE J,CONNELL T O,MARA F O.The ARPA MT evaluation methodologies:evolution,lessons,and future approaches[C]//Proceedings of the First Conference of the Association for Machine Translation in the Americas.1994:193-205.
[22]WISEMAN S,SHIEBER S,RUSH A.Challenges in data-to-document generation[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.2017:2253-2263.
[23]LAVIE A,SAGAE K,JAYARAMAN S.The significance of recall in automatic metrics for MT evaluation[C]//6th Conference of the Association for Machine Translation in the Americas.2004:134-143.
[24]ANDERSON P,FERNANDO B,JOHNSON M,et al.SPICE:semantic propositional image caption evaluation[C]//14th European Conference on Computer Vision.2016:382-398.
[25]BENGIO Y,DUCHARME R,VINCENT P,et al.A neuralprobabilistic language model[J].Journal of Machine Learning Research 3,2003,3(6):1137-1155.
[26]MAJUMDER G,PAKRAY P,GELBUKH A,et al.Semantictextual similarity methods,tools,and applications:a survey[J].Computaciony Sistemas,2016,20(4):647-665.
[27]MIKOLOV T,CHEN K,CORRADO G,et al.Efficient estimation of word representations in vector space[J].arXiv:1301.3781,2013.
[28]WANG Q,XU J G,CHEN H,et al.Two improved continuous bag-of-word models[C]//2017 International Joint Conference on Neural Networks.IEEE,2017:2851-2856.
[29]PENNINGTON J,SOCHER R,MANNING C.Glove:globalvectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Proces-sing.2014:1532-1543.
[30]DEVLIN J,CHANG M,LEE K,et al.Bert:pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics.2019:4171-4186.
[31]JIN X L,ZHANG S W,LIU J.Word semantic similarity calculation based on Word2vec[C]//2018 International Conference on Control,Automation and Information Sciences.IEEE,2018:12-16.
[32]HUANG P S,HE X D,GAO J F,et al.Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM International Conference on Information & Knowledge Management.2013:2333-2338.
[33]SHEN Y L,HE X D,GAO J F,et al.A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval[C]//Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management.2014:101-110.
[34]CER D,DIAB M,AGIRRE E,et al.SemEval-2017 Task 1:semantic textual similarity multilingual and crosslingual focused evaluation[C]//Proceedings of the 11th International Workshop on Semantic Evaluation.2017:1-14.
[35]WANG M,HU F H.The application of NLTK library for python natural language processing in corpus research[J].Theory and Practice in Language Studies,2021,11(9):1041-1049.
[36]BIRD S.NLTK:the natural language toolkit[C]//Proceedings of the COLING/ACL on Interactive Presentation Sessions.2006:69-72.
[1] 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木.
中文预训练模型研究进展
Advances in Chinese Pre-training Models
计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[2] 韩红旗, 冉亚鑫, 张运良, 桂婕, 高雄, 易梦琳.
基于共同子空间分类学习的跨媒体检索研究
Study on Cross-media Information Retrieval Based on Common Subspace Classification Learning
计算机科学, 2022, 49(5): 33-42. https://doi.org/10.11896/jsjkx.210200157
[3] 刘硕, 王庚润, 彭建华, 李柯.
基于混合字词特征的中文短文本分类算法
Chinese Short Text Classification Algorithm Based on Hybrid Features of Characters and Words
计算机科学, 2022, 49(4): 282-287. https://doi.org/10.11896/jsjkx.210200027
[4] 刘凯, 张宏军, 陈飞琼.
基于领域适应嵌入的军事命名实体识别
Name Entity Recognition for Military Based on Domain Adaptive Embedding
计算机科学, 2022, 49(1): 292-297. https://doi.org/10.11896/jsjkx.201100007
[5] 杨杏丽.
分类学习算法的性能度量指标综述
Survey for Performance Measure Index of Classification Learning Algorithm
计算机科学, 2021, 48(8): 209-219. https://doi.org/10.11896/jsjkx.200900216
[6] 杨进才, 曹元, 胡泉, 沈显君.
基于Transformer模型与关系词特征的汉语因果类复句关系自动识别
Relation Classification of Chinese Causal Compound Sentences Based on Transformer Model and Relational Word Feature
计算机科学, 2021, 48(6A): 295-298. https://doi.org/10.11896/jsjkx.200500019
[7] 杨青, 张亚文, 朱丽, 吴涛.
基于注意力机制和BiGRU融合的文本情感分析
Text Sentiment Analysis Based on Fusion of Attention Mechanism and BiGRU
计算机科学, 2021, 48(11): 307-311. https://doi.org/10.11896/jsjkx.201000075
[8] 张玉帅, 赵欢, 李博.
基于BERT和BiLSTM的语义槽填充
Semantic Slot Filling Based on BERT and BiLSTM
计算机科学, 2021, 48(1): 247-252. https://doi.org/10.11896/jsjkx.191200088
[9] 程婧, 刘娜娜, 闵可锐, 康昱, 王新, 周扬帆.
一种低频词词向量优化方法及其在短文本分类中的应用
Word Embedding Optimization for Low-frequency Words with Applications in Short-text Classification
计算机科学, 2020, 47(8): 255-260. https://doi.org/10.11896/jsjkx.191000163
[10] 李舟军,范宇,吴贤杰.
面向自然语言处理的预训练技术研究综述
Survey of Natural Language Processing Pre-training Techniques
计算机科学, 2020, 47(3): 162-173. https://doi.org/10.11896/jsjkx.191000167
[11] 霍丹, 张生杰, 万路军.
基于上下文的情感词向量混合模型
Context-based Emotional Word Vector Hybrid Model
计算机科学, 2020, 47(11A): 28-34. https://doi.org/10.11896/jsjkx.191100114
[12] 景丽, 李曼曼, 何婷婷.
结合扩充词典与自监督学习的网络评论情感分类
Sentiment Classification of Network Reviews Combining Extended Dictionary and Self-supervised Learning
计算机科学, 2020, 47(11A): 78-82. https://doi.org/10.11896/jsjkx.200400061
[13] 杨丹浩,吴岳辛,范春晓.
一种基于注意力机制的中文短文本关键词提取模型
Chinese Short Text Keyphrase Extraction Model Based on Attention
计算机科学, 2020, 47(1): 193-198. https://doi.org/10.11896/jsjkx.181202261
[14] 王乐乐,汪斌强,刘建港,张建辉,苗启广.
基于递归神经网络的恶意程序检测研究
Study on Malicious Program Detection Based on Recurrent Neural Network
计算机科学, 2019, 46(7): 86-90. https://doi.org/10.11896/j.issn.1002-137X.2019.07.013
[15] 李舟军,王昌宝.
基于深度学习的机器阅读理解综述
Survey on Deep-learning-based Machine Reading Comprehension
计算机科学, 2019, 46(7): 7-12. https://doi.org/10.11896/j.issn.1002-137X.2019.07.002
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!