[3E5-GS-2] Machine learning

[3E5-GS-2-01] End-to-end Training of Deep Boltzmann Machines Using Unbiased MCMC

〇Shohei Taniguchi¹, Masahiro Suzuki¹, Yusuke Iwasawa¹, Yutaka Matsuo¹ (1. The University of Tokyo)

Keywords:Boltzmann machine, unbiased MCMC

We address the problem of biased gradient estimation in deep Boltzmann machines (DBMs). The existing
method to obtain an unbiased estimator uses a maximal coupling based on a Gibbs sampler, but when the state
is high-dimensional, it takes a long time to converge. In this study, we propose to use a coupling based on the
Metropolis-Hastings (MH) and to initialize the state around a local mode of the target distribution. Because of
the propensity of MH to reject proposals, the coupling tends to converge in only one step with a high probability,
leading to high efficiency. We find that our method allows DBMs to be trained in an end-to-end fashion without
greedy pretraining. We also propose some practical techniques to further improve the performance of DBMs. We
empirically demonstrate that our training algorithm enables DBMs to show comparable generative performance to
other deep generative models, achieving the FID score of 10.33 for MNIST.