I found this talk to be fascinating. I’ve been a big fan of LIME but never really understood the details of how it works under the hood. I understood that it works on an observation-by-observation basis but I never knew that it permutates data, tests against the black box model, and then builds a simple linear model to explain it. Cool. My notes are below the video.
Notes
Input > black box > output; when don’t understand the black box like neural nets
Example, will the loan default?
Typical classification problem
Loan and applicant information relative to historical data
Linear relationships are easy
Nonlinear relationships via a Decision Tree can still be interpreted
Big data creates more complexity and dimensions
One way to overcome this: use feature importance
Feature importance doesn’t give us any understanding if it’s a linear or nonlinear relationship
Gets better with partial dependence plots
Can’t do partial dependence plots for neural nets
You can create Bayesian Networks / shows dependencies of all variables including output variable and strength of relationship
Bummer: Not as accurate as some other algorithms
Can give you a global understanding but not a detailed explanation
Accuracy vs Interpretability tradeoff. Does it exist?
Enter LIME! Local Interpretable Model-agnostic Explanations
At a local level, it uses a linear model to explain the prediction
Creates an observation, creates fake data (permutation), then calculates a similarity score between the fake and original data, then it takes your black box algo (neural nets?), tries different combinations of predictors
Takes those features with similarity scores, fits a simple model to it to define weights and scores to explain it
Without knowing what the model picks up on if it’s signal or noise. You need LIME to verify!
Can apply to NLP/Text models
Why is it important? Trust / Predict / Improve
LIME helps feature engineering by none ML practitioners
LIME can help comply with GDPR
Understanding our models can help prevent vulnerable people