论文标题
是从头开始建造的代表吗？在语言模型中对本地组成的实证检查
Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models
论文作者
论文摘要
组成性是可以从其组成部分得出的含义的现象，是人类语言的标志。同时，许多短语是非组成的，具有孤立的每个部分的含义。代表这两种类型的短语对于语言理解至关重要，但是现代语言模型（LMS）是否学会这样做是一个悬而未决的问题。在这项工作中，我们研究了这个问题。我们首先提出一个问题，即考虑到其成分的较长短语的LM内部表示。我们发现，鉴于其子女的仿射转变，可以通过某种准确性来预测父词的表示。尽管我们期望预测精度与人类语义组成性的判断相关，但我们发现事实并非如此，这表明LMS可能无法准确区分组成和非组成短语。我们执行各种分析，阐明LMS不同品种何时会产生组成表示，并讨论对未来建模工作的影响。
Compositionality, the phenomenon where the meaning of a phrase can be derived from its constituent parts, is a hallmark of human language. At the same time, many phrases are non-compositional, carrying a meaning beyond that of each part in isolation. Representing both of these types of phrases is critical for language understanding, but it is an open question whether modern language models (LMs) learn to do so; in this work we examine this question. We first formulate a problem of predicting the LM-internal representations of longer phrases given those of their constituents. We find that the representation of a parent phrase can be predicted with some accuracy given an affine transformation of its children. While we would expect the predictive accuracy to correlate with human judgments of semantic compositionality, we find this is largely not the case, indicating that LMs may not accurately distinguish between compositional and non-compositional phrases. We perform a variety of analyses, shedding light on when different varieties of LMs do and do not generate compositional representations, and discuss implications for future modeling work.
