Computer Science

Computer Science ›› 2023, Vol. 50 ›› Issue (11A): 230300002-6.doi: 10.11896/jsjkx.230300002

• Artificial Intelligence • Previous Articles     Next Articles

Joint Method for Spoken Language Understanding Based on BERT and Multiple Feature Gate Mechanism

WANG Zhiming1, ZHENG Kai2   

  1. 1 School of Computer Science,South China Normal University,Guangzhou 510631,China
    2 Network Center,South China Normal University,Guangzhou 510631,China
  • Published:2023-11-09
  • About author:ZHENG Kai,born in 1978,Ph.D,senior engineer.His main research interests include education information technology and network security.
  • Supported by:
    National Natural Science Foundation of China(62237001).

PDF (PC)

473

Abstract: Intent classification and slot filling are two subtasks of spoken language comprehension that are used to identify the intent of text sequences in a conversation system and to obtain slot information from the text sequences that may be used to further infer the exact substance of the intent.Recent research has revealed that these two tasks are connected and can reinforce one another.However,the majority of joint techniques now just use one feature to establish the relationship between the two by only exchanging parameters,which frequently results in issues like poor model generalization and low feature utilization.In order to solve these problems,a novel joint model is proposed that adds an intent feature extraction layer and a slot feature extraction layer for additional text feature extraction based on BERT to improve text vector representation.It also fuses features from different parties using gate mechanism to fully utilize the semantic relationship between the two tasks to predict labels.Experimental fin-dings on the openly accessible datasets ATIS and SNIPS demonstrate the effectiveness of the proposed model in improving intent categorization and slot filling performance,outperforming current approaches.

Key words: Intent classification, Slot filling, Joint learning, Gate mechanism, BERT

CLC Number: 

  • TP391
[1]ZHANG C,LI Y,DUN,et al.Joint Slot Filling and Intent Detection via Capsule Neural Networks[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:5259-5267.
[2]HAIHONG E,NIU P,CHEN Z,et al.A Novel Bi-directionalInterrelated Model for Joint Intent Detection and Slot Filling[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:5467-5471.
[3]LIU B,LANE I.Attention-based recurrent neural network mo-dels for joint intent detection and slot filling[J].arXiv:1609.01454,2016.
[4]GOO C W,GAO G,HSU Y K,et al.Slot-gated modeling forjoint slot filling and intent prediction[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 2(Short Papers).2018:753-757.
[5]QIN L,CHE W,LI Y,et al.A Stack-Propagation Frameworkwith Token-Level Intent Detection for Spoken Language Understanding[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:2078-2087.
[6]ZHANG X,WANG H.A joint model of intent determinationand slot filling for spoken language understanding[C]//IJCAI.2016:2993-2999.
[7]HAFFNER P,TUR G,WRIGHT J H.Optimizing SVMs forcomplex call classification[C]//2003 IEEE International Conference on Acoustics,Speech,and Signal Processing(ICASSP’03).IEEE,2003,1:1.
[8]SCHAPIRE R E,SINGER Y.BoosTexter:A boosting-basedsystem for text categorization[J].Machine Learning,2000,39(2):135-168.
[9]KIM Y.Convolutional Neural Networks for Sentence Classification[C]//EMNLP.2014.
[10]RAVURI S,STOLCKE A.Recurrent neural network and LSTMmodels for lexical utterance classification[C]//Sixteenth Annual Conference of the International Speech Communication Association.2015.
[11]CHO K,VAN MERRIËNBOER B,GULCEHREC,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[J].arXiv:1406.1078,2014.
[12]LAI S,XU L,LIU K,et al.Recurrent convolutional neural networks for text classification[C]//Twenty-ninth AAAI Conference on Artificial Intelligence.2015.
[13]ZHOU P,SHI W,TIAN J,et al.Attention-based bidirectional long short-term memory networks for relation classification[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics(volume 2:Short papers).2016:207-212.
[14]JOHNSON R,ZHANG T.Deep pyramid convolutional neuralnetworks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers).2017:562-570.
[15]RAYMOND C,RICCARDI G.Generative and discriminative algorithms for spoken language understanding[C]//8th Annual Conference of the International Speech Communication Association(INTERSPEECH 2007).2007.
[16]YAO K,ZWEIG G,HWANG M Y,et al.Recurrent neural networks for language understanding[C]//Interspeech.2013:2524-2528.
[17]YAO K,PENG B,ZWEIG G,et al.Recurrent conditional ran-dom field for language understanding[C]//2014 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).IEEE,2014:4077-4081.
[18]YAO K,PENG B,ZHANG Y,et al.Spoken language under-standing using long short-term memory neural networks[C]//2014 IEEE Spoken Language Technology Workshop(SLT).IEEE,2014:189-194.
[19]SIMONNET E,CAMELIN N,DELÉGLISE P,et al.Exploring the use of attention-based recurrent neural networks for spoken language understanding[C]//Machine Learning for Spoken Language Understanding and Interaction NIPS 2015 workshop(SLUNIPS 2015).2015.
[20]JEONG M,LEE G G.Triangular-chain conditional randomfields[J].IEEE Transactions on Audio,Speech,and Language Processing,2008,16(7):1287-1302.
[21]GUO D,TUR G,YIH W,et al.Joint semantic utterance classification and slot filling with recursive neural networks[C]//2014 IEEE Spoken Language Technology Workshop(SLT).IEEE,2014:554-559.
[22]XU P,SARIKAYA R.Convolutional neural network based triangular crf for joint intent detection and slot filling[C]//2013 IEEE Workshop on Automatic Speech Recognition and Understanding.IEEE,2013:78-83.
[23]CHEN S,YU S.Wais:Word attention for joint intent detection and slot filling[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019,33(1):9927-9928.
[24]QIN L,LIU T,CHE W,et al.A co-interactive transformer for joint slot filling and intent detection[C]//2021 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP 2021).IEEE,2021:8193-8197.
[25]CHEN Q,ZHUO Z,WANG W.Bert for joint intent classification and slot filling[J].arXiv:1902.10909,2019.
[26]VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[J].Advances in Neural Information Processing Systems,2017,30.
[27]REIMERS N,GUREVYCH I,REIMERS N,et al.Sentence-BERT:Sentence Embeddings using Siamese BERT-Networks[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.Association for Computational Linguistics,2019:671-688.
[28]CHOI H,KIM J,JOE S,et al.Evaluation of BERT and ALBERT sentence embedding performance on downstream NLP tasks[C]//2020 25th International conference on pattern recognition(ICPR).IEEE,2021:5482-5487.
[29]HE C,CHEN S,HUANG S,et al.Using convolutional neural network with BERT for intent determination[C]//2019 International Conference on Asian Language Processing(IALP).IEEE,2019:65-70.
[30]SAFAYA A,ABDULLATIF M,YURET D.Kuisail at semeval-2020 task 12:Bert-cnn for offensive speech identification in social media[C]//Proceedings of the Fourteenth Workshop on Semantic Evaluation.2020:2054-2059.
[31]TUR G,HAKKANI-TÜR D,HECKL.What is left to be understood in ATIS?[C]//2010 IEEE Spoken Language Technology Workshop.IEEE,2010:19-24.
[32]COUCKE A,SAADE A,BALL A,et al.Snips voice platform:an embedded spoken language understanding system for private-by-design voice interfaces[J].arXiv:1805.10190,2018.
[33]HAKKANI-TÜR D,TÜR G,CELIKYILMAZ A,et al.Multi-domain joint semantic frame parsing using bi-directional rnn-lstm[C]//Interspeech.2016:715-719.
[34]WU D,DING L,LU F,et al.SlotRefine:A fast non-autoregressive model for joint intent detection and slot filling[J].arXiv:2010.02693,2020.
[1] LIU Yingying, YANG Qiuhui, YAO Bangguo, LIU Qiaoyun. Study on REST API Test Case Generation Method Based on Dependency Model [J]. Computer Science, 2023, 50(9): 101-107.
[2] ZHAO Jiangjiang, WANG Yang, XU Yingying, GAO Yang. Extractive Automatic Summarization Model Based on Knowledge Distillation [J]. Computer Science, 2023, 50(6A): 210300179-7.
[3] LI Binghui, FANG Huan, MEI Zhenhui. Interpretable Repair Method for Event Logs Based on BERT and Weak Behavioral Profiles [J]. Computer Science, 2023, 50(5): 38-51.
[4] LUO Liang, CHENG Chunling, LIU Qian, GUI Yaocheng. Answer Selection Model Based on MLP and Semantic Matrix [J]. Computer Science, 2023, 50(5): 270-276.
[5] WANG Yali, ZHANG Fan, YU Zeng, LI Tianrui. Aspect-level Sentiment Classification Based on Interactive Attention and Graph Convolutional Network [J]. Computer Science, 2023, 50(4): 196-203.
[6] DENG Liang, QI Panhu, LIU Zhenlong, LI Jingxin, TANG Jiqiang. BGPNRE:A BERT-based Global Pointer Network for Named Entity-Relation Joint Extraction Method [J]. Computer Science, 2023, 50(3): 42-48.
[7] LIU Zhe, YIN Chengfeng, LI Tianrui. Chinese Spelling Check Based on BERT and Multi-feature Fusion Embedding [J]. Computer Science, 2023, 50(3): 282-290.
[8] JIA Jingdong, ZHANG Minnan, ZHAO Xiang, HUANG Jian. Study on Scheduling Algorithm of Intelligent Order Dispatching [J]. Computer Science, 2023, 50(11A): 230300029-7.
[9] YANG Jincai, MA Chen, XIAO Ming. Relation Recognition of Unmarked Complex Sentences Based on Feature Fusion [J]. Computer Science, 2023, 50(11A): 221100065-6.
[10] QIAN Taiyu, CHEN Yifei, PANG Bowen. Audit Text Named Entity Recognition Based on MacBERT and Adversarial Training [J]. Computer Science, 2023, 50(11A): 230200083-6.
[11] LIN Zhengchao, LI Bicheng. Aspect-based Sentiment Analysis Based on Local Context Focus Mechanism and Talking-Head Attention [J]. Computer Science, 2023, 50(11A): 220900266-6.
[12] WANG Ming, PENG Jian, HUANG Fei-hu. Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction [J]. Computer Science, 2022, 49(8): 40-48.
[13] YU Jia-qi, KANG Xiao-dong, BAI Cheng-cheng, LIU Han-qing. New Text Retrieval Model of Chinese Electronic Medical Records [J]. Computer Science, 2022, 49(6A): 32-38.
[14] KANG Yan, WU Zhi-wei, KOU Yong-qi, ZHANG Lan, XIE Si-yu, LI Hao. Deep Integrated Learning Software Requirement Classification Fusing Bert and Graph Convolution [J]. Computer Science, 2022, 49(6A): 150-158.
[15] YU Ben-gong, ZHANG Zi-wei, WANG Hui-ling. TS-AC-EWM Online Product Ranking Method Based on Multi-level Emotion and Topic Information [J]. Computer Science, 2022, 49(6A): 165-171.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.