Maximum Likelihood from Incomplete Data Via the EM Algorithm

Inside the paper

The problem

When observations leave some information missing, maximizing their likelihood directly can be awkward even when the corresponding complete-data calculation is straightforward. Different applications had produced iterative solutions that were not always recognized as instances of the same approach.

The idea

Treat the unobserved information as part of a larger data set and alternate between averaging over its possible values and updating the parameters. Each iteration uses the current parameter estimate to decide how to average.

How it works

• Choose a complete-data model whose possible outcomes are compatible with the observations and initialize its parameters.
• In the expectation step, calculate the expected complete-data log likelihood conditional on the observed data, using the current parameters.
• In the maximization step, choose parameters that maximize that expected log likelihood, then repeat the two steps.
• For mixture models, the missing information can be the component responsible for each observation, so the expectation step assigns conditional component probabilities.

What it showed

The paper proves that the observed-data likelihood does not decrease under its EM iterations and studies conditions for convergence. It also sets out examples ranging from missing values and mixtures to factor analysis.

Read it for

Start with Section 2 for the definition and Section 4.3 for the mixture example; Section 4.1 gives another concrete setting. Skim the more general convergence arguments on a first pass, then return to them to understand what monotonic improvement does and does not guarantee.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.