How StockTwits Uses Machine Learning to Make Better Products

Fascinating behind the scenes interview of StockTwit’s Senior Data Scientist Garrett Hoffman. He shares great tidbits on how StockTwits uses machine learning for sentiment analysis. I’ve summarized the highlights below:

• Idea generation is a huge barrier for active trading
• Next gen of traders uses social media to make decisions
• Garrett solves data problems and builds features for the StockTwits platform
• This includes: production data science, product analytics, and insights research
• Understanding social dynamics makes for a better user experience
• Focus is to understand social dynamics of StockTwits (ST) community
• Focuses on what’s happening inside the ST community
• ST’s market sentiment model helps users with decision making
• Users ’tag’ content for bullish or bearish classes
• Only 20 to 30% of content is tagged
• Using ST’s market sentiment model increases coverage to 100%
• For Data Science work, Python Stack is used
• Use: Numpy, SciPy, Pandas, Scikit-Learn
• Jupyter Notebooks for research and prototyping
• Flask for API deployment
• For Deep Learning, uses Tensorflow with AWS EC2 instances
• Can spin up GPU’s as needed
• Deep Learning methods used are Recurrent Neural Nets, Word2Vec, and Autoencoders
• Stays abreast of new machine learning techniques from blogs, conferences and Twitter
• Follows Twitter accounts from Google, Spotify, Apple, and small tech companies
• One area ST wants to improve on is DevOps around Data Science
• Bridge the gap between research/prototype phase and embedding it into tech stack for deployment
• Misconception that complex solutions are best
• Complexity ONLY ok if it leads to deeper insight
• Simple solutions are best
• Future long-term ideas: use AI around natural language