论文标题
样本外模型评估的诊断工具
Diagnostic Tool for Out-of-Sample Model Evaluation
论文作者
论文摘要
评估模型健身是机器学习的关键部分。标准范式是通过最大程度地减少培训数据的所选损失函数来学习模型，以实现未来数据的少量损失。在本文中，我们考虑使用有限的校准数据集来表征未来，模型的样本外损失。我们提出了一个简单的模型诊断工具，该工具在弱假设下提供有限样本保证。该工具易于计算和解释。提出了几个数值实验，以显示所提出的方法如何量化分布移位的影响，有助于回归分析，并启用模型选择以及高参数调整。
Assessment of model fitness is a key part of machine learning. The standard paradigm is to learn models by minimizing a chosen loss function averaged over training data, with the aim of achieving small losses on future data. In this paper, we consider the use of a finite calibration data set to characterize the future, out-of-sample losses of a model. We propose a simple model diagnostic tool that provides finite-sample guarantees under weak assumptions. The tool is simple to compute and to interpret. Several numerical experiments are presented to show how the proposed method quantifies the impact of distribution shifts, aids the analysis of regression, and enables model selection as well as hyper-parameter tuning.
