High-Resolution Image Synthesis with Latent Diffusion Models

Inside the paper

The problem

Diffusion models can produce detailed images, but repeatedly running a denoising network over full-resolution pixels is expensive. The authors ask how much of that work can happen in a smaller representation without losing useful visual detail.

The idea

First learn to compress and reconstruct images, then train diffusion in the compressed representation. The paper also adds a way to condition generation on inputs such as text or spatial layouts.

How it works

• An autoencoder maps an image to a spatial latent representation and reconstructs pixels from it; perceptual and adversarial losses help preserve visually important structure.
• A diffusion network learns to predict noise added to the encoded images at different noise levels, working in the smaller latent space.
• At generation time, the model denoises a latent sample and the decoder turns the resulting representation into an image.
• Cross-attention lets the denoising network incorporate conditioning signals encoded by a separate model, including text descriptions.

What it showed

The paper compares compression choices, sampling costs and image quality across several generation tasks. It presents text-conditioned images as well as results for tasks such as inpainting and super-resolution, showing the range tested rather than just unconditional synthesis.

Read it for

Begin with Figure 2 and Sections 3.1–3.3 to see how compression, diffusion and conditioning fit together. Section 4 provides the comparisons; detailed architecture and training settings in the supplement can wait until implementation matters.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.