论文标题
通过交替的教学来减轻语言模型中的意外记忆
Mitigating Unintended Memorization in Language Models via Alternating Teaching
论文作者
论文摘要
最近的研究表明，语言模型倾向于记住培训语料库中的稀有或独特序列，从而可以泄漏用户数据的敏感属性。我们采用教师培训框架，并提出了一种称为交替教学的新方法，以减轻顺序建模中的意外记忆。在我们的方法中，对多个教师进行了培训，他们希望在每个时间步骤中以交替的方式来监督学生模型的培训，并预测教师的隐私培训集培训。 Librispeech数据集上的实验表明，所提出的方法比其他对应物获得了卓越的隐私性结果。与没有预防意外记忆的预防相比，当训练记录足够时，总体公用事业损失很小。
Recent research has shown that language models have a tendency to memorize rare or unique sequences in the training corpora which can thus leak sensitive attributes of user data. We employ a teacher-student framework and propose a novel approach called alternating teaching to mitigate unintended memorization in sequential modeling. In our method, multiple teachers are trained on disjoint training sets whose privacy one wishes to protect, and teachers' predictions supervise the training of a student model in an alternating manner at each time step. Experiments on LibriSpeech datasets show that the proposed method achieves superior privacy-preserving results than other counterparts. In comparison with no prevention for unintended memorization, the overall utility loss is small when training records are sufficient.
