Visual Instruction Tuning

Inside the paper

The problem

Language assistants could follow instructions, but adapting them to answer questions about images required suitable multimodal instruction data. Human-written image conversations were relatively scarce.

The idea

The authors use a text-only language model to generate instruction-following examples from image descriptions and annotations, then use those examples to train a visual assistant called LLaVA.

How it works

• Image captions and bounding-box descriptions are supplied to a language model as a textual representation of the scene.
• The language model generates conversations, detailed descriptions, and reasoning examples grounded in that textual scene information.
• A pretrained visual encoder extracts image features, which a learned projection maps into the language model’s input space.
• The model is trained on image and instruction pairs to produce the corresponding assistant responses, using the generated data.

What it showed

The paper evaluates the resulting assistant on a constructed multimodal instruction-following test and on the Science QA benchmark. It reports promising results while also discussing limitations in the generated data and evaluation.

Read it for

Table 1 and Section 3 explain how the examples are made; Figure 1 and Section 4 show how image features enter the language model. Read Section 5 for the evaluation, and skim the prompt templates initially if you want the main mechanism first.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.