Abstract: The faithfulness of text summaries,which refers to their factual consistency with the original texts,is very important for the practical application of automatic text summarization.Current methods for evaluating the faithfulness of summaries have shortcomings in utilizing text summarization datasets,and the constructed unfaithful summaries differ significantly from the original texts,which limit the effectiveness of these evaluation methods.To solve this problem,this paper proposes a summary faithfulness evaluation model,FaithEval,based on data augmentation and two-stage training.Firstly,two data augmentation methods are defined:Similarity Search with Same Topic and Insert and Fill External Mask,which are used to generate summaries that are related but not faithful to the original texts.These methods are used to extract training data from the text summarization dataset.Secondly,to fully utilize the dataset information,the model is trained in two stages based on the training data constructed from the original texts and the reference summaries,progressively strengthening the faithfulness evaluation ability of the model.Finally,the test set for summary faithfulness evaluation SFETS,is constructed manually to provide a benchmark for testing model performance.Experiments show that FaithEval performs well on both SFETS and Rank19 datasets,and achieves the current state-of-the-art performance on the SFETS dataset.

Key words: Text summarization,Faithfulness evaluation,Data augmentation,Two-stage training,Benchmark test set