论文标题
对比句子表示学习的可区分数据增强
Differentiable Data Augmentation for Contrastive Sentence Representation Learning
论文作者
论文摘要
通过大量未标记的句子或标记的句子对通过对比度学习框架进行微调训练的语言模型是获得高质量句子表示的常见方法。尽管对比度学习框架已经表明了其在句子表示学习上的优越性，而不是以前的方法，但由于它用于构造正面对的简单方法，此类框架的潜力尚未探索。以此为动机，我们提出了一种从原始培训示例中获得艰难积极性的方法。我们方法的关键要素是使用预训练的语言模型附加到的前缀，该前缀允许在对比度学习过程中进行可区分的数据增强。我们的方法可以通过两个步骤进行总结：有监督的前缀调整，然后使用未标记或标记的示例进行联合对比度进行微调。我们的实验证实了我们数据增强方法的有效性。所提出的方法对半监督和监督设置下的现有方法产生了重大改进。我们在低标记的数据设置下进行的实验还表明，我们的方法比最先进的对比度学习方法更有效。
Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sentence pairs is a common way to obtain high-quality sentence representations. Although the contrastive learning framework has shown its superiority on sentence representation learning over previous methods, the potential of such a framework is under-explored so far due to the simple method it used to construct positive pairs. Motivated by this, we propose a method that makes hard positives from the original training examples. A pivotal ingredient of our approach is the use of prefix that is attached to a pre-trained language model, which allows for differentiable data augmentation during contrastive learning. Our method can be summarized in two steps: supervised prefix-tuning followed by joint contrastive fine-tuning with unlabeled or labeled examples. Our experiments confirm the effectiveness of our data augmentation approach. The proposed method yields significant improvements over existing methods under both semi-supervised and supervised settings. Our experiments under a low labeled data setting also show that our method is more label-efficient than the state-of-the-art contrastive learning methods.
