A Unified Approach to Interpreting Model Predictions

Inside the paper

The problem

Complex prediction models can be difficult to inspect, and existing methods for explaining individual outputs did not make their relationships or differences obvious.

The idea

Treat an explanation as a simpler model that adds feature contributions, then use properties of that model to identify a particular allocation of credit for a prediction.

How it works

• Represent a case using simplified features that can be present or absent, and express the explanation as a baseline plus a contribution for each present feature.
• Require the explanation to match the prediction at the case, assign no credit to missing features, and respect increases in a feature’s contribution.
• Calculate the resulting values by averaging each feature’s marginal effect over possible subsets of the other features.
• Approximate these values with weighted regression for general models or with procedures adapted to particular model structures.

What it showed

The paper relates several existing explanation methods through the additive framework and derives a unique attribution under its stated properties. Its experiments compare approximations on computation, human judgments and explanations of image classes.

Read it for

Read Sections 2 and 3 for the shared representation and its properties, then Section 4.1 for the general approximation. Figure 5 illustrates explanations across image classes; the model-specific details in Section 4.2 can wait.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.