Abstract: Pretraining data detection determines whether a candidate text has been memorized by a large language model(LLM) during pretraining.Existing mainstream methods employ fixed aggregation rules that select tokens via hard thresholds or keyword rules and then apply equal-weight averaging,and they rely on token surprisal alone while ignoring the reference gap.To address these limitations,existing methods are unified as special cases of a weighted token aggregation framework,and a soft-weighted aggregation method named SoftTail is proposed.A lightweight multilayer perceptron(MLP) with only 17 parameters gene-rates continuous weights for all tokens and incorporates the reference gap(the log-probability advantage of the target model over a smaller reference model of the same family) as a complementary signal to surprisal.Over 175 fold cross-validations on the MIMIR benchmark,SoftTail achieves the highest values on both macro-averaged area under the curve(AUC) and true positive rate at a 5% false positive rate(TPR@5%FPR),with the latter yielding a 4.77 percentage points improvement over Min-K%++;a paired bootstrap test confirms statistical significance.Ablation study results show that introducing the reference gap is the key driver of performance gains.The detection advantage remains consistent on the WikiMIA benchmark and under GPT-2 cross-architecture validation.

Key words: Pretraining data detection, Membership inference, Soft-weighted aggregation, Reference gap, Token surprisal