Meet the Practitioner: Otis Anderson
Otis has a phenomenally eclectic background. He's a social scientist by training (Econ) and has done everything from Affirmative Action consulting to Human Resources Analytics to leading Data Science teams at cutting edge startups. He's currently at OpenDoor, working on their financial models and product performance. Otis and I worked together very closely for two years at Clover and I found him to have a unique perspective on how you generate value with data and math, especially if you come from a non-traditional background.
Otis' key takeaway:
“Using lots of numbers without the accompanying critical thinking is actually bad.”
Tell us your story:
What attracted you to data science as a career? Baseball analytics actually drove me into Data Science. I wanted to be able to argue more coherently for my beliefs and understanding data turned out to be the path into that. Once I was doing this for baseball, I started to realized that making sense of numbers and data was a super power for economics, politics, and policy- other things that are near and dear to my heart. I learned statistics in service of this. I've always been drawn more to the humanities side, chasing questions that are meaningful to the human experience, rather doing quantitative things as an end in and of themselves.
How did you break in? My first job after I graduated was in Affirmative Action consulting, where I put that skill set to good use. I found myself really attracted to the analysis side, both because of its importance and the categorical messiness inherent in the data. In the real world data never stands on its own, rather you have to find and communicate an interpretation that makes sense. I found I had real talent at doing this- turning stats into stories so that good decisions could be made. My coworkers noticed this (as well as my penchant for debates) and suggested that I consider going back to school for a masters. My company gave you nice raise when you got an advanced degree, so I pursued a master's. I'm not sure I would recommend this for everyone, but it worked out well for me. I got to double down on the things I enjoyed- dealing with messy data and math, and building up narratives that helped the reader gain an understanding of what was going on, as opposed to just looking at numbers.
The masters and affirmative action gig got me into Google HR, where I mainly worked on the ATS system. I got to do some data cleaning that contributed to the results Laszlo Bock is publicizing in his book. That got me to Yammer as one of the earliest product analysts when someone in my network got hired there which put me at the top of their LinkedIn searches. I helped build out the team there and built the foundations for analyzing the product performance and was there through the Microsoft acquisition. That took me to Clover, where I led the Data Science and Product Analytics teams, which took me to OpenDoor. There was a lot of serendipity in my progression, and I'm very happy with where I've landed.
Tell us about your professional persona:
What is your primary expertise? I think of my primary expertise as identifying, answering, and communicating the answer to questions around causality. There are lots of questions that are important to a business that are of the nature “Why does this happen?” or “How can we make something else happen?”. I've been very effective at breaking these down into answerable questions and communicating the answers to those questions in ways people can understand and internalize:
- What do we know?
- What do we think we can do?
- Why do we think we can do it?
Essentially moving a company from desires to evidence to plans to outcomes.
What skills did you have to develop to get there? To execute on my expertise, I've had to learn to build trust with my audience. This led me to develop the following:
- Strong intuition around problem spaces- how to locate relevant information and ignore noise, how to identify the core elements of a problem, and how to ensure instrumented measurement loops. This comes with experience, but is a skill that can be developed.
- SQL and iterations. I don't rely on complex models to achieve my results. I prefer to focus on speed of delivery and making the iteration loop as fast as possible. SQL is the best tool I've found for this (although I'm always open to folks showing me others).
- Experiment design. Learning how to build good test environments and experiments that yield good answers.
- Building good user/customer models. I care a lot about the interface for the work I do- everything I do needs to be understood and acted upon by another human. I've invested a lot in developing frameworks for how users and customers interact with analysis and information, which has helped my ability to deliver results.
Tell us about your journey:
What were the hardest things you needed to pick up/learn? I've always been a bit behind on Tech, so I've tended to default to tools I already know. I don't have great avenues for learning about new libraries or pieces of software, which is a thing I've been working to get better at. But the single hardest thing I've found to learn is conscientiousness- making sure that my analyses are error free, that bug rates are low in my code, so that my results can be relied upon. This wasn't something that came naturally to me. I've had to fix it with a combination of hard work (focused labor) and by surrounding myself with systems that would catch these faults before they became problems (eg. engineers who would build testing frameworks I could use).
What turned out to be easier than you thought? Math. Not that it's not needed, but deep math turned out to be less necessary that I originally thought. I found critical thinking and intuition to be just as valuable, and that I could partner with others when I found myself needing more horsepower. As an example of this, while at Microsoft I suspected that Bing was not randomizing users correctly during A/B tests. I could see hints of it in the data but couldn't prove it. Another person on my team with a formal math background was able to rigorously demonstrate the bias.
What advice would you give those looking to take their game to the next level? (eg. What should they spend their time on?) I generally see two types of folks entering the field: Math People (Physicists, Statisticians, Engineers, and the like) and Social Scientists (Economists, Psychologists, etc.). I have different things for each type. Math People: Think about the human part of the problem- the decisions that the users need to make that you are trying to affect with your products. Great algorithms are not enough- you need to model from a human point of view, understand the human foibles that lead them to bad decisions, and the limits of your users that may cause them to misunderstand your work. Social Scientists: Math is important and you will need reps. Practice as much as you can, and don't worry about being the smartest kid in the room. You'll have a lot of formalism to catch up on. Practice will get you there.
Last words: Tell us something you believed strongly in where you turned out to be wrong
That's an interesting question. I don't tend to hold on to my beliefs very strongly, but one spot that sticks out is a dissonance I had between my academic training and commercial work. While at Yammer I worried a lot about the external validity problem, whether the results in our experiments would be valid from customer to customer. I basically took the stand that the experiment was a valid assessment of the feature added, and had no value beyond that. This fundamentalist view caused me to not lean on experimentation aggressively as a systematic method for learning about the business. It turned out that in the real world experimentation was the best tool we had and we should have pushed much much harder on using the results to make generalizations about our users. Moreover, we could have monitored the performance we were getting on a go forward basis to ensure that the results we had gotten were valid. Basically, I learned that in a way having high standards for truth can cause you to not push the results that you have as far as you should.