LoRA: Low-Rank Adaptation of Large Language Models

Inside the paper

The problem

Adapting a large pretrained language model separately for many tasks can require storing a full set of changed weights for each one. Full fine-tuning also makes all those weights part of the training workload.

The idea

LoRA leaves the pretrained weights fixed and represents a task-specific weight change as the product of small matrices. The resulting update has low rank and can be added to the original weight matrix.

How it works

• Select weight matrices in the pretrained transformer to adapt and retain their original values throughout training.
• Introduce a pair of trainable matrices whose product has the same shape as each selected weight matrix but a much smaller intermediate dimension.
• Initialize the adaptation so its initial contribution is zero, then train the new matrices on the downstream task.
• For deployment, add the learned matrix product to the frozen weight, avoiding a separate added computation in the forward pass.

What it showed

Across the paper’s language-model experiments, LoRA performs competitively with full fine-tuning and other adaptation baselines while training fewer parameters. Its rank experiments also find that small ranks can work well on the tasks studied, though the authors do not claim this holds for every task.

Read it for

Start with Section 4 and Figure 1 for the construction, then read the comparison tables in Section 5. Section 7 examines rank and matrix choices; the longer baseline descriptions can be skimmed initially.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.