基于BERT和多特征门控机制的口语理解联合方法

doi:10.11896/jsjkx.230300002

摘要/Abstract

摘要： 意图分类和槽位填充是口语理解任务的两个子任务,用于在对话系统中识别文本序列的意图以及从文本序列中提取出能进一步确定意图具体内容的槽位信息。近些年的研究表明,这两个任务具有相关性,且可以相互促进。然而,现有的大多数联合方法仅利用单一特征简单地通过共享参数建立两者之间的关联,而这往往会导致模型泛化能力差及特征利用程度低等问题。针对这些问题,提出了一种新的联合模型。模型在BERT的基础上引入了意图特征提取层和槽位特征提取层用于进一步提取文本特征,以增强文本向量表示能力,并通过多特征门机制融合了多方特征,充分利用两个任务之间的语义关系预测标签。在公开数据集ATIS和SNIPS上的实验结果表明,提出的模型能有效提升意图分类和槽位填充的性能,相比已有方法取得了更优的结果。

关键词: 意图分类, 槽位填充, 联合学习, 门控机制, BERT

Abstract: Intent classification and slot filling are two subtasks of spoken language comprehension that are used to identify the intent of text sequences in a conversation system and to obtain slot information from the text sequences that may be used to further infer the exact substance of the intent.Recent research has revealed that these two tasks are connected and can reinforce one another.However,the majority of joint techniques now just use one feature to establish the relationship between the two by only exchanging parameters,which frequently results in issues like poor model generalization and low feature utilization.In order to solve these problems,a novel joint model is proposed that adds an intent feature extraction layer and a slot feature extraction layer for additional text feature extraction based on BERT to improve text vector representation.It also fuses features from different parties using gate mechanism to fully utilize the semantic relationship between the two tasks to predict labels.Experimental fin-dings on the openly accessible datasets ATIS and SNIPS demonstrate the effectiveness of the proposed model in improving intent categorization and slot filling performance,outperforming current approaches.

Key words: Intent classification, Slot filling, Joint learning, Gate mechanism, BERT

中图分类号:

TP391

王志明, 郑凯. 基于BERT和多特征门控机制的口语理解联合方法[J]. 计算机科学, 2023, 50(11A): 230300002-6. https://doi.org/10.11896/jsjkx.230300002

WANG Zhiming, ZHENG Kai. Joint Method for Spoken Language Understanding Based on BERT and Multiple Feature Gate Mechanism[J]. Computer Science, 2023, 50(11A): 230300002-6. https://doi.org/10.11896/jsjkx.230300002

参考文献

[1]ZHANG C,LI Y,DUN,et al.Joint Slot Filling and Intent Detection via Capsule Neural Networks[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:5259-5267.
[2]HAIHONG E,NIU P,CHEN Z,et al.A Novel Bi-directionalInterrelated Model for Joint Intent Detection and Slot Filling[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:5467-5471.
[3]LIU B,LANE I.Attention-based recurrent neural network mo-dels for joint intent detection and slot filling[J].arXiv:1609.01454,2016.
[4]GOO C W,GAO G,HSU Y K,et al.Slot-gated modeling forjoint slot filling and intent prediction[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 2(Short Papers).2018:753-757.
[5]QIN L,CHE W,LI Y,et al.A Stack-Propagation Frameworkwith Token-Level Intent Detection for Spoken Language Understanding[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:2078-2087.
[6]ZHANG X,WANG H.A joint model of intent determinationand slot filling for spoken language understanding[C]//IJCAI.2016:2993-2999.
[7]HAFFNER P,TUR G,WRIGHT J H.Optimizing SVMs forcomplex call classification[C]//2003 IEEE International Conference on Acoustics,Speech,and Signal Processing(ICASSP’03).IEEE,2003,1:1.
[8]SCHAPIRE R E,SINGER Y.BoosTexter:A boosting-basedsystem for text categorization[J].Machine Learning,2000,39(2):135-168.
[9]KIM Y.Convolutional Neural Networks for Sentence Classification[C]//EMNLP.2014.
[10]RAVURI S,STOLCKE A.Recurrent neural network and LSTMmodels for lexical utterance classification[C]//Sixteenth Annual Conference of the International Speech Communication Association.2015.
[11]CHO K,VAN MERRIËNBOER B,GULCEHREC,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[J].arXiv:1406.1078,2014.
[12]LAI S,XU L,LIU K,et al.Recurrent convolutional neural networks for text classification[C]//Twenty-ninth AAAI Conference on Artificial Intelligence.2015.
[13]ZHOU P,SHI W,TIAN J,et al.Attention-based bidirectional long short-term memory networks for relation classification[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics(volume 2:Short papers).2016:207-212.
[14]JOHNSON R,ZHANG T.Deep pyramid convolutional neuralnetworks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers).2017:562-570.
[15]RAYMOND C,RICCARDI G.Generative and discriminative algorithms for spoken language understanding[C]//8th Annual Conference of the International Speech Communication Association(INTERSPEECH 2007).2007.
[16]YAO K,ZWEIG G,HWANG M Y,et al.Recurrent neural networks for language understanding[C]//Interspeech.2013:2524-2528.
[17]YAO K,PENG B,ZWEIG G,et al.Recurrent conditional ran-dom field for language understanding[C]//2014 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).IEEE,2014:4077-4081.
[18]YAO K,PENG B,ZHANG Y,et al.Spoken language under-standing using long short-term memory neural networks[C]//2014 IEEE Spoken Language Technology Workshop(SLT).IEEE,2014:189-194.
[19]SIMONNET E,CAMELIN N,DELÉGLISE P,et al.Exploring the use of attention-based recurrent neural networks for spoken language understanding[C]//Machine Learning for Spoken Language Understanding and Interaction NIPS 2015 workshop(SLUNIPS 2015).2015.
[20]JEONG M,LEE G G.Triangular-chain conditional randomfields[J].IEEE Transactions on Audio,Speech,and Language Processing,2008,16(7):1287-1302.
[21]GUO D,TUR G,YIH W,et al.Joint semantic utterance classification and slot filling with recursive neural networks[C]//2014 IEEE Spoken Language Technology Workshop(SLT).IEEE,2014:554-559.
[22]XU P,SARIKAYA R.Convolutional neural network based triangular crf for joint intent detection and slot filling[C]//2013 IEEE Workshop on Automatic Speech Recognition and Understanding.IEEE,2013:78-83.
[23]CHEN S,YU S.Wais:Word attention for joint intent detection and slot filling[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019,33(1):9927-9928.
[24]QIN L,LIU T,CHE W,et al.A co-interactive transformer for joint slot filling and intent detection[C]//2021 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP 2021).IEEE,2021:8193-8197.
[25]CHEN Q,ZHUO Z,WANG W.Bert for joint intent classification and slot filling[J].arXiv:1902.10909,2019.
[26]VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[J].Advances in Neural Information Processing Systems,2017,30.
[27]REIMERS N,GUREVYCH I,REIMERS N,et al.Sentence-BERT:Sentence Embeddings using Siamese BERT-Networks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.Association for Computational Linguistics,2019:671-688.
[28]CHOI H,KIM J,JOE S,et al.Evaluation of BERT and ALBERT sentence embedding performance on downstream NLP tasks[C]//2020 25th International conference on pattern recognition(ICPR).IEEE,2021:5482-5487.
[29]HE C,CHEN S,HUANG S,et al.Using convolutional neural network with BERT for intent determination[C]//2019 International Conference on Asian Language Processing(IALP).IEEE,2019:65-70.
[30]SAFAYA A,ABDULLATIF M,YURET D.Kuisail at semeval-2020 task 12:Bert-cnn for offensive speech identification in social media[C]//Proceedings of the Fourteenth Workshop on Semantic Evaluation.2020:2054-2059.
[31]TUR G,HAKKANI-TÜR D,HECKL.What is left to be understood in ATIS?[C]//2010 IEEE Spoken Language Technology Workshop.IEEE,2010:19-24.
[32]COUCKE A,SAADE A,BALL A,et al.Snips voice platform:an embedded spoken language understanding system for private-by-design voice interfaces[J].arXiv:1805.10190,2018.
[33]HAKKANI-TÜR D,TÜR G,CELIKYILMAZ A,et al.Multi-domain joint semantic frame parsing using bi-directional rnn-lstm[C]//Interspeech.2016:715-719.
[34]WU D,DING L,LU F,et al.SlotRefine:A fast non-autoregressive model for joint intent detection and slot filling[J].arXiv:2010.02693,2020.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed