Fraud Analytics in RapidMiner
This is a great video presentation on Fraud Analytics use case with RapidMiner. See my notes below.
##Some key concepts
- More complex model, the lower the training error but higher test error.
- Simple models are better, try explaining them to children.
- Data Scientists understand the technical aspect, need to communicate results with analysts.
- Sell results to businesses. Tie $ to the results.
- Speak same language with business. Map performance metrics to business related figures.
- AUC and recall doesn’t necessarily mean $ to the business, show how.
- A/B testing method widely used in Marketing. Also consider a “do nothing” model and compare with implementing data science solution.
- Don’t fear sharing best practices and ideas with similar businesses.
- Fraud model follows traditional validation method. 80% Training and 20% as Holdout.
- Both training/holdout sets taken across same time period.
- Handy trick, use sum of transactions as example weights. (this is cool)
- Apply $ value to your true/false positive/negatives.
- Compare with Default model (no model).
- Generate a money plot!
##Some questions
- How does this relate to regression?
- If a simple model not good enough, how do you sell a complex one like Deep Learning?
- Is it better to have the Data Science team be embedded in the Business Unit or as a separate team?
- How do you try to explain the uncertainty of prediction intervals to business stakeholders?
- How do you account for seasonal drift?
- The model will drift overtime, should the model be updated or retrained over time?
- Do you build a model to optimize business results or is it a byproduct of the prediction?