论文标题
通过即时汇编和矢量化启用快速差异私有SGD
Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization
论文作者
论文摘要
私人机器学习中的一个常见疼痛点是执行差异私有随机梯度下降（DPSGD）时产生的重要运行时开销，这可能与两个数量级一样大。我们彻底地证明，通过利用强大的语言原语，包括矢量化，即时编译和静态图优化，可以大大减少这些间接费用，在许多情况下，几乎与最佳的非私人运行时间匹配。这些收益在两个框架中实现：JAX和TensorFlow。 JAX通过XLA编译器为这些原语作为语言的核心功能提供了丰富的支持。我们还重建了TensorFlow隐私的核心部分，集成了TensorFlow 2以及XLA编译的功能，对当前版本版本进行了重大内存和运行时的改进。与最佳替代方案相比，这些方法使我们可以实现多达50倍的加速。我们的代码可在https://github.com/thesalon/fast-dpsgd上找到。
A common pain point in differentially private machine learning is the significant runtime overhead incurred when executing Differentially Private Stochastic Gradient Descent (DPSGD), which may be as large as two orders of magnitude. We thoroughly demonstrate that by exploiting powerful language primitives, including vectorization, just-in-time compilation, and static graph optimization, one can dramatically reduce these overheads, in many cases nearly matching the best non-private running times. These gains are realized in two frameworks: JAX and TensorFlow. JAX provides rich support for these primitives as core features of the language through the XLA compiler. We also rebuild core parts of TensorFlow Privacy, integrating features from TensorFlow 2 as well as XLA compilation, granting significant memory and runtime improvements over the current release version. These approaches allow us to achieve up to 50x speedups in comparison to the best alternatives. Our code is available at https://github.com/TheSalon/fast-dpsgd.
