Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Inside the paper

The problem

A language generator stores knowledge in its parameters, making it difficult to inspect or update individual facts. Earlier retrieval systems and generative models offered different ways to answer knowledge-intensive questions.

The idea

Combine a document retriever with a sequence generator, treating retrieved passages as evidence for the answer. The model can train the retrieval and generation components together on downstream examples.

How it works

• A dense retriever encodes a question and finds passages with similar representations in a prebuilt index.
• A generator receives the question along with a retrieved passage and assigns probabilities to possible output sequences.
• One variant uses a single passage for an entire output, while another can combine evidence from different passages for individual tokens.
• Training sums over retrieved candidates when estimating the likelihood of the target, adjusting the retriever and generator together.

What it showed

The paper reports strong results on open-domain question answering and also evaluates answer generation, question generation and fact verification. In its question generation analysis, evaluators judge the retrieved model’s outputs more specific than those of a parametric baseline.

Read it for

Read Sections 2.1–2.4 for the two model variants and training, then Section 4.1 for the question answering results. The task-specific setup in Section 3 can be consulted as needed.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.