Concrete Problems in AI Safety

Inside the paper

The problem

An autonomous system can cause an accident while pursuing the objective its designers gave it, particularly when that objective leaves important details unspecified.

The idea

This paper turns a broad concern about accidents into a set of technical questions, using a cleaning robot to make each failure mode concrete and suggesting research directions.

How it works

• For side effects, ask how an agent can avoid unnecessary disruption without receiving a separate penalty for every possible kind of damage.
• For reward hacking, examine how an agent might exploit the measurement of a goal instead of achieving what its designer intended.
• For scalable oversight, consider how to learn appropriate behavior when human evaluation is useful but too expensive to provide constantly.
• For safe exploration and distributional shift, study how to limit harmful experiments and recognize situations unlike those encountered during training.

What it showed

The paper offers problem formulations, examples, links to prior work and candidate approaches across these areas. It is a research agenda rather than an experimental demonstration that the proposed approaches solve the problems.

Read it for

Read Section 2 for the robot examples, then choose among Sections 3–7 according to the failure mode that interests you. The related-work survey in Section 8 can be saved for a later pass.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.