论文标题
基于变压器的变异自动编码器的准符号语义几何形状
Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncoder
论文作者
论文摘要
正式/符号语义可以通过其\ textit {bertization}或\ textit {composition}属性为句子表示提供规范，严格的可控性和解释性。我们如何将这种属性交付到当前的分布句子表示形式中以控制和解释语言模型（LMS）的产生？在这项工作中，我们从理论上将句子语义构图为\ textit {语义角色 - 单词content}特征的组成，并提出正式的语义几何形状。为了将这种几何形状注入基于变压器的LMS（即GPT2），我们采用监督方法部署了基于变压器的变异自动编码器，在这种方法中，可以在低维潜在高斯空间中操纵和解释句子的生成。此外，我们提出了一种新的探测算法来指导句子向量在这种几何上的运动。实验结果表明，正式的语义几何形状可以潜在地为句子产生提供更好的控制和解释。
Formal/symbolic semantics can provide canonical, rigid controllability and interpretability to sentence representations due to their \textit{localisation} or \textit{composition} property. How can we deliver such property to the current distributional sentence representations to control and interpret the generation of language models (LMs)? In this work, we theoretically frame the sentence semantics as the composition of \textit{semantic role - word content} features and propose the formal semantic geometry. To inject such geometry into Transformer-based LMs (i.e. GPT2), we deploy Transformer-based Variational AutoEncoder with a supervision approach, where the sentence generation can be manipulated and explained over low-dimensional latent Gaussian space. In addition, we propose a new probing algorithm to guide the movement of sentence vectors over such geometry. Experimental results reveal that the formal semantic geometry can potentially deliver better control and interpretation to sentence generation.
