论文标题
在具有增量解析和动态甲骨文的语言模型中明确建模语法
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle
论文作者
论文摘要
语法对于我们对语言的思考至关重要。未能捕获输入语言的结构可能导致概括问题和过度参数化。在目前的工作中，我们提出了一种新的语法意识语言模型：句法有序记忆（SOM）。该模型用增量解析器明确对结构进行建模，并保持标准语言模型（从左到右）的条件概率设置。为了训练增量解析器并避免暴露偏见，我们还提出了一种新型的动态甲骨文，因此SOM对错误的解析决策更加可靠。实验表明，SOM可以在语言建模，增量解析和句法概括测试中获得强大的结果，而使用的参数少于其他模型。
Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing and syntactic generalization tests, while using fewer parameters than other models.
