[2B6-GS-2] Machine learning: Basics / Theory

[2B6-GS-2-02] Representation of Denoising Autoencoders by Tweedie's formula and its Analysis

〇Yuta Aishima¹, Kazushi Ikeda¹ (1. Nara Institute of Science and Technology)

Keywords:denoising autoencoder, exponential family, Tweedie's formula, feature extraction

Denoising autoencoders learn the score of the data-generating distribution, i.e., ∇log p(x). However, theoretical studies have discussed only the case when the corruption process is Gaussian. To generalize the previous results, in this study, we extend the class of distributions to an exponential family. By using Tweedie’s formula, we show that the generalized DAE learns score of marginal distribution q(˜x) = ∫ q(˜x|x)p(x)dx, while Gaussian DAE learns the score of the data-generating distribution p(x). Then, we focus on the encoder of denoisng autoenoders and investigate what this encoder learn. Numerical experiment shows that this encoder extract features which relate the shape of the data-generating distribution.