“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

Inside the paper

The problem

A classifier can score well overall while relying on misleading signals for particular decisions, making it hard to decide when to trust a prediction or a model.

The idea

Approximate the classifier near one example with a small interpretable model, and use explanations from representative examples to inspect the classifier more broadly.

How it works

• Express an example using understandable components, such as words in a document or contiguous regions of an image, even if the classifier uses a different representation.
• Create nearby variants by removing components, and query the original classifier to obtain predictions for those variants.
• Fit a sparse linear model to those predictions, weighting variants by their closeness to the example so the explanation describes local behavior.
• Select a diverse collection of explanations that covers important components, giving a person a compact view of model behavior across examples.

What it showed

Experiments found that local explanations recovered important features in controlled settings and helped people compare models and improve a text classifier. An image case study exposed reliance on background cues in a wolf-versus-husky task.

Read it for

Begin with Sections 3.1–3.4 and Figure 3 for the local procedure, then Section 6 and Figure 11 for the human studies. The optimization details of the representative-example selection in Section 4 can be skimmed initially.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.