论文标题
蒙版更多，然后掩盖：通过解开[蒙版]令牌，有效地预读了掩盖语言模型
Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token
论文作者
论文摘要
蒙面语言模型（MLMS）的预训练会消耗大量计算，以在下游NLP任务上取得良好的结果，从而产生了较大的碳足迹。在Vanilla MLM中，虚拟令牌（蒙版）充当占位符，并从未掩盖的令牌中收集上下文化信息以恢复损坏的信息。它提出了一个问题，即我们是否可以在较晚的一层附加[掩码]，以减少早期层的序列长度并使预训练更有效。我们显示：（1）[掩码]确实可以将s附加在以后的一层中，从嵌入一词中解开； （2）可以使用几层从未掩盖的令牌中收集上下文化信息。通过进一步将掩蔽率从15％提高到50％，我们可以将Roberta-Base和Roberta-Large从头开始，只有78％和68％的原始计算预算，而不会在胶水基准上进行任何退化。当预算预算预先培训时，我们的方法在8个胶水任务中的6个胶水中超过了罗伯塔，平均要花0.4％。
The pre-training of masked language models (MLMs) consumes massive computation to achieve good results on downstream NLP tasks, resulting in a large carbon footprint. In the vanilla MLM, the virtual tokens, [MASK]s, act as placeholders and gather the contextualized information from unmasked tokens to restore the corrupted information. It raises the question of whether we can append [MASK]s at a later layer, to reduce the sequence length for earlier layers and make the pre-training more efficient. We show: (1) [MASK]s can indeed be appended at a later layer, being disentangled from the word embedding; (2) The gathering of contextualized information from unmasked tokens can be conducted with a few layers. By further increasing the masking rate from 15% to 50%, we can pre-train RoBERTa-base and RoBERTa-large from scratch with only 78% and 68% of the original computational budget without any degradation on the GLUE benchmark. When pre-training with the original budget, our method outperforms RoBERTa for 6 out of 8 GLUE tasks, on average by 0.4%.
