Latent Dirichlet Allocation

Inside the paper

The problem

A collection of documents contains recurring patterns of words, but the themes and their proportions in each document are not observed directly. Earlier models of document collections provided useful representations, while leaving questions about how to model new documents.

The idea

Model each document as a mixture of latent topics and each topic as a distribution over words. A probability distribution over document mixtures makes it possible to describe previously unseen documents in the same model.

How it works

• For each document, draw a distribution over topics from a Dirichlet distribution.
• For each word position, draw a topic from the document's mixture and then draw a word from that topic's word distribution.
• Approximate the otherwise intractable posterior over hidden topic assignments and document mixtures with a simpler variational distribution.
• Fit model parameters by alternating approximate inference for documents with parameter updates across the collection.

What it showed

The paper evaluates the model on held-out document prediction, text classification and collaborative filtering. It reports that topic-based representations can provide useful features for classification while reducing the representation's dimensionality.

Read it for

Start with Section 3 and its graphical model, then read Section 6 for a worked example. Section 5 explains approximate inference; Section 7 provides the experiments, and its individual applications can be read separately.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.