论文标题
超越单个模型的持续学习
Continual Learning Beyond a Single Model
论文作者
论文摘要
持续学习的越来越多的研究集中在灾难性的遗忘问题上。尽管已经尝试缓解这个问题，但大多数方法都在持续学习设置中采用单个模型。在这项工作中，我们质疑这一假设，并表明采用集合模型可以是一种简单而有效的方法来提高持续性能。但是，随着模型数量的增加，合奏的培训和推理成本可能会大大增加。在这种局限性的激励下，我们研究了不同的集合模型，以了解他们在不断学习的情况下的好处和缺点。最后，为了克服合奏的高计算成本，我们利用神经网络子空间的最新进展提出了一种具有与单个模型相似的计算廉价算法，但享受了合奏的性能优势。
A growing body of research in continual learning focuses on the catastrophic forgetting problem. While many attempts have been made to alleviate this problem, the majority of the methods assume a single model in the continual learning setup. In this work, we question this assumption and show that employing ensemble models can be a simple yet effective method to improve continual performance. However, ensembles' training and inference costs can increase significantly as the number of models grows. Motivated by this limitation, we study different ensemble models to understand their benefits and drawbacks in continual learning scenarios. Finally, to overcome the high compute cost of ensembles, we leverage recent advances in neural network subspace to propose a computationally cheap algorithm with similar runtime to a single model yet enjoying the performance benefits of ensembles.
