论文标题
无监督解析的分层变压器
A Hierarchical Transformer for Unsupervised Parsing
论文作者
论文摘要
自然语言的基本结构是分层的。单词结合到短语中，这又形成了从句。对这种层次结构的认识可以帮助机器学习模型执行许多语言任务。但是，大多数这样的模型只是按顺序处理文本，并且对学习层次结构编码的层次结构没有偏见。在本文中，我们通过使其能够学习层次表示形式来扩展最近的变压器模型（Vaswani等，2017）。为了实现这一目标，我们适应了Shen等人，2018年引入的订购机制，以适应变压器结构的自发模块。我们在语言建模上训练新模型，然后将其应用于无监督解析的任务。我们在WSJ10数据集的免费可用子集中取得了合理的结果，F1得分约为50％。
The underlying structure of natural language is hierarchical; words combine into phrases, which in turn form clauses. An awareness of this hierarchical structure can aid machine learning models in performing many linguistic tasks. However, most such models just process text sequentially and there is no bias towards learning hierarchical structure encoded into their architecture. In this paper, we extend the recent transformer model (Vaswani et al., 2017) by enabling it to learn hierarchical representations. To achieve this, we adapt the ordering mechanism introduced in Shen et al., 2018, to the self-attention module of the transformer architecture. We train our new model on language modelling and then apply it to the task of unsupervised parsing. We achieve reasonable results on the freely available subset of the WSJ10 dataset with an F1-score of about 50%.
