论文标题
如何使用信息理论选择目标函数
How to select an objective function using information theory
论文作者
论文摘要
在机器学习或科学计算中，通过目标函数来衡量模型性能。但是，为什么选择一个目标而不是另一个目标呢？信息理论给出一个答案：为了最大化模型中的信息，请选择代表最少位中误差的目标函数。要评估不同的目标，请将它们转换为可能性功能。作为可能性，它们的相对幅度表示我们应该多么强烈地偏爱一个目标而不是另一个目标，而该关系的日志代表了其比特长度的差异以及其不确定性的差异。换句话说，更喜欢将任何客观最小化的不确定性最小化。在信息理论范式下，最终目标是最大化信息（并最大程度地减少不确定性），而不是任何特定的效用。我们认为，这种范式非常适合具有多种用途且没有明确效用的模型，例如用于了解气候变化效果的大型地球系统模型。
In machine learning or scientific computing, model performance is measured with an objective function. But why choose one objective over another? Information theory gives one answer: To maximize the information in the model, select the objective function that represents the error in the fewest bits. To evaluate different objectives, transform them into likelihood functions. As likelihoods, their relative magnitude represents how strongly we should prefer one objective versus another, and the log of that relation represents the difference in their bit-length, as well as the difference in their uncertainty. In other words, prefer whichever objective minimizes the uncertainty. Under the information-theoretic paradigm, the ultimate objective is to maximize information (and minimize uncertainty), as opposed to any specific utility. We argue that this paradigm is well-suited to models that have many uses and no definite utility, like the large Earth system models used to understand the effects of climate change.
