关键词: 自然语言处理, 文档级关系抽取, DocRED, 交叉多头注意力, 解耦知识蒸馏, 远程监督数据, KL散度

Abstract: Document-level relation extraction is an important research direction in the field of natural language processing,aiming to extract semantic relationships between entities from unstructured or semi-structured natural language documents.This paper proposes a solution combining decoupled knowledge distillation and cross multi-head attention mechanisms to address the DocRE task.Firstly,the cross multi-head attention mechanism can not only simultaneously focus on elements in different attention heads,enabling the model to exchange and integrate information at different granularities and levels but also allow the model to consider the correlation between head and tail entities and their relations when calculating attention,thereby enhancing the model'sunderstanding of complex relationships and improving the learning of entity feature representations.Additionally,to further optimize the model's performance,this paper introduces a decoupled knowledge distillation method to adapt to distantly supervised data.This method decouples the original KL divergence loss into target class knowledge distillation loss(TCKDL) and non-target class knowledge distillation loss(NCKDL),which can adjust their weight importance through hyperparameters,increasing the flexibility and effectiveness of the knowledge distillation process.Particularly,it enables more precise knowledge transfer and learning when dealing with noise in the DocRED distantly supervised data.Experimental results show that the proposed model can more effectively extract relationships between entity pairs on the DocRED dataset.

Key words: Natural language processing, Document-Level relation extraction, DocRED, Cross Multi-head attention, Decoupled knowledge distillation, Distantly supervised data, Kullback-Leibler divergence