论文标题
相对位置预测作为文本编码的预培训
Relative Position Prediction as Pre-training for Text Encoders
论文作者
论文摘要
含义由其所保留的公司定义。但是，公司是两个方面：它基于令牌的身份以及其位置（拓扑）。我们认为，以职位为中心的观点更加通用和有用。 NLP中的经典MLM和CLM目标很容易被用作整个词汇的位置预测。调整编码NLP中范式的相对位置以创建用于自学学习的相对标签，我们试图展示由下游任务上的性能来判断的优越的预训练。
Meaning is defined by the company it keeps. However, company is two-fold: It's based on the identity of tokens and also on their position (topology). We argue that a position-centric perspective is more general and useful. The classic MLM and CLM objectives in NLP are easily phrased as position predictions over the whole vocabulary. Adapting the relative position encoding paradigm in NLP to create relative labels for self-supervised learning, we seek to show superior pre-training judged by performance on downstream tasks.
