Abstract: Recent advancements in Large Language Models (LLMs) have yielded remarkable success across diverse fields.However,handling long contexts remains a significant challenge for LLMs due to the quadratic time and space complexity of attention mechanisms and the growing memory consumption of the key-value cache during generation.To address this issue,this paper proposes MemLong’a memory-augmented method for long-text modeling,which enhances long-context language modeling by leveraging an external retriever to access historical information.MemLong integrates a non-parametric retrieval-memory module with a partially trainable large language model,and introduces a fine-grained,controllable retrieval attention mechanism that effectively utilizes semantically relevant text blocks.The non-parametric module is responsible for retrieving relevant historical information from an external knowledge base,while the LLM generates outputs by fusing this retrieved information with the current input.The proposed attention mechanism allows the model to dynamically adjust its focus on the retrieved information during generation.Comprehensive evaluations on multiple long-context language modeling benchmarks demonstrate that MemLong consistently outperforms other state-of-the-art LLMs.Furthermore,MemLong significantly enhances the model’s capacity to process long texts.On a single NVIDIA 3090 GPU,MemLong can scale the effective context length from 4 000 to 80 000 tokens,representing a 20-fold increase.This breakthrough enables MemLong to process longer input texts,leading to a better understanding and ge-neration of long-form content.It provides new possibilities for tackling ultra-long text tasks and opens up promising new directions for future research in long-text language modeling.

Key words: Retrieval augment language modeling, Long context generation, Natural language processing, Long context evaluation, Retrieval augment generation