Fascinating behind the scenes interview of StockTwit’s Senior Data Scientist Garrett Hoffman. He shares great tidbits on how StockTwits uses machine learning for sentiment analysis. I’ve summarized the highlights below:

• Idea generation is a huge barrier for active trading
• Next gen of traders uses social media to make decisions
• Garrett solves data problems and builds features for the StockTwits platform
• This includes: production data science, product analytics, and insights research
• Understanding social dynamics makes for a better user experience
• Focus is to understand social dynamics of StockTwits (ST) community
• Focuses on what’s happening inside the ST community
• ST’s market sentiment model helps users with decision making
• Users ’tag’ content for bullish or bearish classes
• Only 20 to 30% of content is tagged
• Using ST’s market sentiment model increases coverage to 100%
• For Data Science work, Python Stack is used
• Use: Numpy, SciPy, Pandas, Scikit-Learn
• Jupyter Notebooks for research and prototyping
• Flask for API deployment
• For Deep Learning, uses Tensorflow with AWS EC2 instances
• Can spin up GPU’s as needed
• Deep Learning methods used are Recurrent Neural Nets, Word2Vec, and Autoencoders
• Stays abreast of new machine learning techniques from blogs, conferences and Twitter
• Follows Twitter accounts from Google, Spotify, Apple, and small tech companies
• One area ST wants to improve on is DevOps around Data Science
• Bridge the gap between research/prototype phase and embedding it into tech stack for deployment
• Misconception that complex solutions are best
• Complexity ONLY ok if it leads to deeper insight
• Simple solutions are best
• Future long-term ideas: use AI around natural language

• AI Machine Learning