论文标题
增强的多教老师选择以进行知识蒸馏
Reinforced Multi-Teacher Selection for Knowledge Distillation
论文作者
论文摘要
在自然语言处理（NLP）任务中，慢速推理速度和GPU使用中的巨大足迹仍然是在生产中应用预训练的深层模型的瓶颈。作为模型压缩的一种流行方法，知识蒸馏将知识从一个或多个大型（教师）模型转移到小型（学生）模型。当多个教师模型进行蒸馏时，最先进的方法在整个蒸馏中为教师模型分配了固定的重量。此外，大多数现有方法分配给每个教师模型的权重。在本文中，我们观察到，由于培训示例的复杂性和学生模型能力的差异，从教师模型中学习的差异可以使学生模型的表现更好。我们系统地开发了一种增强方法，将权重动态分配给教师模型以进行不同的培训实例，并优化学生模型的表现。我们对几个NLP任务的广泛实验结果清楚地验证了我们方法的可行性和有效性。
In natural language processing (NLP) tasks, slow inference speed and huge footprints in GPU usage remain the bottleneck of applying pre-trained deep models in production. As a popular method for model compression, knowledge distillation transfers knowledge from one or multiple large (teacher) models to a small (student) model. When multiple teacher models are available in distillation, the state-of-the-art methods assign a fixed weight to a teacher model in the whole distillation. Furthermore, most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled. We systematically develop a reinforced method to dynamically assign weights to teacher models for different training instances and optimize the performance of student model. Our extensive experimental results on several NLP tasks clearly verify the feasibility and effectiveness of our approach.
