Learning to Predict by the Methods of Temporal Differences

Inside the paper

The problem

In a prediction problem that unfolds over time, evidence arrives before the final outcome is known. A conventional supervised update waits for that outcome and treats earlier observations as separate examples.

The idea

Temporal-difference methods learn from changes between successive predictions, so they can update before the final result arrives. A parameter controls how strongly a change affects earlier observations.

How it works

• At each observation, produce a prediction of the eventual outcome using adjustable weights and the available features.
• Compare the new prediction with the preceding one, treating their difference as a signal for updating earlier predictions.
• Maintain a trace of past prediction gradients, with older contributions weighted down according to the chosen parameter.
• At the end of a sequence, use the actual outcome as the final prediction so that the last difference incorporates what happened.

What it showed

The paper relates one member of the family to the supervised rule under per-sequence updates and gives convergence and optimality results for restricted cases. Its examples show that using intermediate predictions can improve learning speed in the settings studied.

Read it for

Read Sections 2.2–2.3 for the construction and Section 3 for the example, then turn to Section 4 for the assumptions behind the theorems. The broader connections in Section 6 can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.