论文标题
贝特的层面指导培训：学习逐渐完善的文档表示形式
Layer-wise Guided Training for BERT: Learning Incrementally Refined Document Representations
论文作者
论文摘要
尽管BERT被NLP社区广泛使用，但对其内部运作知之甚少。已经尝试了几次尝试阐明伯特的某些方面，这常常与结论相矛盾。一个非常引起的关注集中在伯特的过度参数化和利用不足的问题上。为此，我们提出了O新颖的方法以结构化的方式进行微调。具体而言，我们专注于大型多标签文本分类（LMTC），其中为文档分配了一个或多个标签，来自一组大型预定义的层次结构构造的标签。我们的方法指导特定的BERT层以预测特定层次结构级别的标签。通过两个LMTC数据集进行实验，我们表明，这种结构化的微调方法不仅会产生更好的分类结果，而且还会带来更好的参数利用。
Although BERT is widely used by the NLP community, little is known about its inner workings. Several attempts have been made to shed light on certain aspects of BERT, often with contradicting conclusions. A much raised concern focuses on BERT's over-parameterization and under-utilization issues. To this end, we propose o novel approach to fine-tune BERT in a structured manner. Specifically, we focus on Large Scale Multilabel Text Classification (LMTC) where documents are assigned with one or more labels from a large predefined set of hierarchically organized labels. Our approach guides specific BERT layers to predict labels from specific hierarchy levels. Experimenting with two LMTC datasets we show that this structured fine-tuning approach not only yields better classification results but also leads to better parameter utilization.
