论文标题
神经面膜生成器：学习为语言模型适应生成自适应单词掩码
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation
论文作者
论文摘要
我们提出了一种方法，以自动生成给定文本的域和任务自适应掩蔽，以进行自我监督的预训练，以便我们可以有效地使语言模型适应特定的目标任务（例如，问题答案）。具体而言，我们提出了一个基于强化学习的新颖框架，该框架可以学习掩盖策略，以便使用生成的面具进一步预培训目标语言模型有助于改善对看不见的文本的任务绩效。我们使用熵批评和熵正则化和经验重播进行强化学习，并提出了一个基于变压器的策略网络，该网络可以考虑给定文本中单词的相对重要性。我们使用Bert和Distilbert作为语言模型来验证几个问题回答和文本分类数据集的神经掩模生成器（NMG），它通过自动学习最佳自适应掩码来胜过基于规则的掩盖策略。
We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.
