论文标题
实体按日期锁定：LMS对看不见的实体的了解
Entity Cloze By Date: What LMs Know About Unseen Entities
论文作者
论文摘要
语言模型（LMS）通常在大型语料库上进行一次培训，并且使用了多年而没有被更新。但是，在一个充满活力的世界中，新实体不断出现。我们提出了一个框架，以分析LMS可以推断出在鉴定LMS时不存在的新实体的框架。我们得出了一个由其起源日期索引的实体数据集，并与他们的英文Wikipedia文章配对，我们可以从中找到有关每个实体的句子。我们评估了LMS对这些句子中蒙面跨度的困惑。我们表明，模型更多地了解实体，例如那些可以访问其文本定义的实体，在此基准测试中实现了较低的困惑。我们的实验结果表明，对LMS的新实体进行推断仍然很难。鉴于其对实体知识和时间索引的广泛报道，我们的数据集可用于评估旨在修改或扩展其知识的LMS和技术。我们的自动数据收集管道可以轻松地用于不断更新我们的基准测试。
Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. However, in a dynamic world, new entities constantly arise. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity. We evaluate LMs' perplexity on masked spans within these sentences. We show that models more informed about the entities, such as those with access to a textual definition of them, achieve lower perplexity on this benchmark. Our experimental results demonstrate that making inferences about new entities remains difficult for LMs. Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge. Our automatic data collection pipeline can be easily used to continually update our benchmark.
