Auto-Encoding Variational Bayes

Inside the paper

The problem

Latent-variable models can describe how observations arise from hidden causes, but their posterior distributions and likelihoods can be hard to calculate. Repeating costly inference for every example also makes learning on large datasets difficult.

The idea

The paper trains a probabilistic encoder and decoder together using a sampled approximation to a variational objective. A change in how latent samples are drawn lets gradients pass through the sampling step.

How it works

• The encoder maps each observation to a distribution over possible latent codes, while the decoder maps a code to a distribution over observations.
• For a Gaussian code, the model transforms independent noise using the encoder’s predicted mean and scale, making the sampled code differentiable with respect to those predictions.
• The objective combines a reconstruction term with a penalty for moving the approximate posterior away from the prior.
• Random minibatches and stochastic gradients update the encoder and decoder together; after training, the encoder can infer codes without a separate iterative procedure.

What it showed

On MNIST and Frey Face, the authors compare learning curves and estimated likelihoods with alternatives including wake-sleep. Their plots also show a learned low-dimensional image manifold; these experiments illustrate the method rather than proving it is best for every latent-variable model.

Read it for

Start with Sections 2.2–2.4 and Algorithm 1 for the objective and sampling construction, then read Section 3 for the concrete autoencoder. Skim the distribution-specific appendix on a first pass and return to Section 5 for the comparisons.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.