论文标题
通过挑战数据暴露关系提取模型的浅启发式方法
Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data
论文作者
论文摘要
收集和注释培训数据的过程可能会引入分布伪像，这可能会限制模型学习正确的概括行为的能力。我们确定了在Tacred训练的SOTA关系提取（RE）模型的失败模式，我们将其归因于数据注释过程中的局限性。我们收集并注释了一个挑战设定，我们称为“挑战性RE”（CRE），基于自然发生的示例，以基准这一行为。我们对四个最先进模型的实验表明，它们确实采用了浅启发式方法，这些启发式方法并未推广到挑战集数据。此外，我们发现尽管总体表现较差，但替代问题回答建模的性能比挑战集中的SOTA模型要好得多。通过添加一些挑战数据作为培训示例，该模型的性能得到了改善。最后，我们提供了有关如何改善RE数据收集以减轻此行为的具体建议。
The process of collecting and annotating training data may introduce distribution artifacts which may limit the ability of models to learn correct generalization behavior. We identify failure modes of SOTA relation extraction (RE) models trained on TACRED, which we attribute to limitations in the data annotation process. We collect and annotate a challenge-set we call Challenging RE (CRE), based on naturally occurring corpus examples, to benchmark this behavior. Our experiments with four state-of-the-art RE models show that they have indeed adopted shallow heuristics that do not generalize to the challenge-set data. Further, we find that alternative question answering modeling performs significantly better than the SOTA models on the challenge-set, despite worse overall TACRED performance. By adding some of the challenge data as training examples, the performance of the model improves. Finally, we provide concrete suggestion on how to improve RE data collection to alleviate this behavior.
