Computer Science ›› 2025, Vol. 52 ›› Issue (6): 324-329.doi: 10.11896/jsjkx.240800017

• Artificial Intelligence • Previous Articles     Next Articles

Research on Hybrid Retrieval-augmented Dual-tower Model

GAO Hongkui, MA Ruixiang, BAO Qihao, XIA Shaojie, QU Chongxiao   

  1. The 52nd Research Institute of China Electronics Technology Group Corporation,Hangzhou 311100,China
  • Received:2024-08-05 Revised:2024-09-26 Online:2025-06-15 Published:2025-06-11
  • About author:GAO Hongkui,born in 1988,postgra-duate.His main research interests include large-scale models in decision-making fields and technologies for intelligent gaming and planning.

Abstract: In the vanguard of knowledge retrieval,particularly in scenarios involving large language models(LLMs),research emphasis has shifted toward employing pure vector retrieval techniques for efficient capture of pertinent information.This information is then fed into large language models for comprehensive distillation and summarization.However,the limitations of this approach lie in its potential inability to fully encompass the intricacies of retrieval through vector representations alone,coupling with an absence of effective ranking mechanisms.This often leads to an overabundance of irrelevant information,thereby diluting the alignment between the final response and the user's actual needs.To address this conundrum,this paper introduces a hybrid retrieval-augmented dual-tower model.This model innovatively integrates a multi-path recall strategy,ensuring that the retrieval results are both comprehensive and highly relevant through complementary recall mechanisms.Architecturally,it adopts a dual-la-yer structure,combining bidirectional recurrent neural networks with text convolutional neural networks.This allows the model to perform multi-level ranking optimization on retrieval results,significantly enhancing the relevance and the precision of top-ranking outcomes.Moreover,the high-quality information,efficiently ranked,is integrated with the original query and fed into a large language model.This exploits the model's deep analytical capabilities to generate more accurate and credible responses.Experimental findings affirm that the proposed method effectively improves retrieval accuracy and system performance overall,markedly enhancing the precision and practicality of large language models in real-world applications.

Key words: Knowledge search, Large language models, Vector retrieval technology, Hybrid retrieval-augmented dual-tower model, Multi-path recall strategy

CLC Number: 

  • TP391
[1]OUYANG L,WU J,JIANG X,et al.Training language models to follow instructions with human feedback[J].Advances in Neural Information Processing Systems,2022,35:27730-27744.
[2]ACHIAM J,ADLER S,AGARWAL S,et al.GPT-4 technicalreport[J].arXiv:2303.08774,2023.
[3]GAO Y,XIONG Y,GAO X,et al.Retrieval-augmented generation for large language models:A survey[J].arXiv:2312.10997,2023.
[4]LEWIS P,PEREZ E,PIKTUS A,et al.Retrieval-augmentedgeneration for knowledge-intensive NLP tasks[C]//Procee-dings of the 34th International Conference on Neural Information Processing Systems.Red Hook,NY:Curran Associates Inc.,2020:9459-9474.
[5]YORAN O,WOLFSON T,RAM O,et al.Making retrieval-augmented language models robust to irrelevant context[J].arXiv:2310.01558,2023.
[6]YU W,ZHANG H PAN,X,et al.2023.Chain-of-note:Enhancing robustness in retrieval-augmented language models[C]//Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.ACL,2024:14672-14685.
[7]LUO Y,YANG Z,MENG F,et al.An empirical study of catastrophic forgetting in large language models during continual fine-tuning[J].arXiv:2308.08747,2023.
[8]IZACARD G,GRAVE E.Leveraging passage retrieval withgenerative models for open do-main question answering[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics:Main Volume.ACL,2021:874-880.
[9]KARPUKHIN V,OGUZ B,MIN S,et al.Dense passage retrie-val for open-domain question answering[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.ACL,2020:6769-6781.
[10]SHI F,CHEN X,MISRA K,et al.Large language models can be easily distracted by irrelevant context[C]//Proceedings of the 40th International Conference on Machine Learning.JMLR,2023:31210-31227.
[11]GAO T,YEN H,YU J,et al.Enabling large language models to generate text with citations[J].arXiv:2305.14627,2023.
[12]ASAI A,WU Z,WANG Y,et al.Self-rag:Learning to retrieve,generate,and critique through self-reflection[J].arXiv:2310.11511,2023.
[13]PRESS O,ZHANG M,MIN S,et al.Measuring and narrowing the compositionality gap in language models[J].arXiv:2210.03350,2022.
[14]XU S,PANG L,SHEN H,et al.Search-in-the-chain:To-wards the accurate,credible and traceable contengeneration for complex knowledge-intensive tasks[J].arXiv:2304.14732,2023.
[15]DHULIAWALA S,KOMEILI M,XU J,et al.Chain-of-verification reduces hallucination in large language modelss[J].arXiv:2309.11495,2023.
[16]CHEN J,XIAO S,ZHANG P,et al.BGE M3-embedding:Multi-lingual,multi-functionality,multi-granularity text embeddings through self-knowledge distillation[J].arXiv:2402.03216,2024.
[17]ROBERTSON S,ZARAGOZA H.The probabilistic relevanceframework:BM25 and beyond[J].Foundations and Trends in Information Retrieval,2009,3(4):333-389.
[18]MIHALCEA R,TARAU P:TextRank:Bringing order into text[C]//Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.ACL,2024:404-411.
[19]ZOU L,ZHANG S,CAI H,et al.Pre-trained language model based ranking in baidu search[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.New York:ACM,2021:4014-4022.
[20]FAN Y,XIE X,CAI Y,et al.Pre-training methods in information retrieval[J].Foundations and Trends in Information Retrieval,2022,16(3):178-317.
[21]HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural Computation,1997,9(8):1735-1780.
[22]KIM Y.Convolutional Neural Networks for Sentence Classification[J].arXiv:1408.5882,2014.
[23]BAI J,BAI S,CHU Y,et al.Qwen technical report[J].arXiv:2309.16609,2023.
[24]RAJPURKAR P,ZHANG J,LOPYREV K,et al.SQuAD:100 000+ questions for machine comprehension of text[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.ACL,2016:2383-2392.
[1] LI Bo, MO Xian. Application of Large Language Models in Recommendation System [J]. Computer Science, 2025, 52(6A): 240400097-7.
[2] HU Caishun. Study on Named Entity Recognition Algorithms in Audit Domain Based on Large LanguageModels [J]. Computer Science, 2025, 52(6A): 240700190-4.
[3] DUN Jingbo, LI Zhuo. Survey on Transmission Optimization Technologies for Federated Large Language Model Training [J]. Computer Science, 2025, 52(1): 42-55.
[4] LI Tingting, WANG Qi, WANG Jiakang, XU Yongjun. SWARM-LLM:An Unmanned Swarm Task Planning System Based on Large Language Models [J]. Computer Science, 2025, 52(1): 72-79.
[5] CHENG Zhiyu, CHEN Xinglin, WANG Jing, ZHOU Zhongyuan, ZHANG Zhizheng. Retrieval-augmented Generative Intelligence Question Answering Technology Based on Knowledge Graph [J]. Computer Science, 2025, 52(1): 87-93.
[6] LIU Yumeng, ZHAO Yijing, WANG Bicong, WANG Chao, ZHANG Baomin. Advances in SQL Intelligent Synthesis Technology [J]. Computer Science, 2024, 51(7): 40-48.
[7] LI Zhanqi, WU Xinwei, ZHANG Lei, LIU Quanzhou, XIE Hui, XIONG Deyi. Automatic Test Case Generation Method for Automotive Electronic Control System Verification [J]. Computer Science, 2024, 51(12): 63-70.
[8] ZHAO Yue, HE Jinwen, ZHU Shenchen, LI Congyi, ZHANG Yingjie, CHEN Kai. Security of Large Language Models:Current Status and Challenges [J]. Computer Science, 2024, 51(1): 68-71.
[9] . [J]. Computer Science, 2007, 34(3): 162-164.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!