Axiomatic Attribution for Deep Networks

Inside the paper

The problem

It is difficult to tell whether an explanation of a neural network prediction is misleading because of the explanation method or because of the model itself.

The idea

The authors start from properties they want an attribution method to satisfy, then construct an explanation by accumulating gradients between a reference input and the example being explained.

How it works

• Choose a baseline input that represents the absence of the signal of interest, such as a black image for an image model.
• Move along the straight line from the baseline to the example, evaluating the model’s output gradient at points along that path.
• For each input feature, average its gradients along the path and multiply by the difference between its actual and baseline values.
• Approximate the integral with a finite sum, and check that the feature contributions approximately add up to the output difference from the baseline.

What it showed

The paper proves properties of the method, including its accounting of the output difference and invariance to equivalent implementations. It applies the method to image, text and chemistry networks, including examples used to investigate model behavior.

Read it for

Read Sections 2 and 3 for the axioms and construction, then Figure 2 and Section 6 for applications. Section 4’s uniqueness argument and its proofs can wait for a more mathematical reading.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.