I’ve spoken to over a hundred AI companies as part of my job at MissingLink.ai and the result of analyzing their experiment, data and compute workflows. Certain challenges were a common theme across many teams – the solutions to these is a concept we call “DeepOps”, deep learning operations. AI promises to revolutionize many aspects of our lives, but many AI companies are struggling under the burden of their massive data and compute resource needs. No data scientist enjoys reinventing the wheel of AI infrastructure. If you’d like to learn about these challenges and successful solutions — join me at #ODSC – Open Data Science where I’ll be talking about DeepOps: Building an AI First Company.
You see, back in 2006 when I got my first real tech job, I was using Visual Source Safe.
And the first time I lost a few code changes, my frustration resulted in writing a Python script that backed up all my source files into a zip file in a folder.
C:\code_backups\2006–09–01-dev.zip
C:\code_backups\2006–09–03-dev.zip
C:\code_backups\2006–10–23-dev.zip
C:\code_backups\2006–11–03-dev.zip
This was pretty darn useless. But at the time, it felt safer than with SourceSafe. When our team adopted SVN, I was worried at first, but it worked reliably and TortoiseSVN was a pleasure to use. I still remember people on the team at the time begrudging any kind of source control.
> We have a network share with all the code, this source control is broken and slowing me down…
Today we know that version control is a critical part of any software team. But it wasn’t always that obvious. Joel Spolsky had to publish a “test” to indicate how good a software team is and the number one question was:
> 1. Do you use source control?
That’s where data science is today. The tools are a mess, some companies are taking it seriously, many aren’t. It’s rare to find teams that version control their data, even though it is one of their most valuable assets. Data scientists are spending 30% of their time building infrastructure. I don’t want to spoil too much, just that the talk delves into exactly these idiosyncrasies, and how the best AI teams are tackling them.
For your notes, here’s the full talk abstract:
Data scientists spend 30% of their time building shoddy infrastructure. Our data shows that many AI teams can accelerate their progress by 10x at least. Deep Learning brings with it enormous amounts of data, complicated experiment results and intense compute requirements. Decades of experience in moving code to production yielded best practices in engineering that have not yet found their place in deep learning teams. Breaking silos to foster trust, a transparent culture, and shared responsibility — we introduce DeepOps — deep learning ops. A set of methodologies, tools and culture where data engineers and scientists collaborate to build a faster and more reliable deep learning pipeline.
https://odsc.com/training/portfolio/deepops-building-an-ai-first-company
I’ve never been to Boston. But I do like a song about how no one knows my name there.
Let’s meet up and chat at ODSC East 2019 🙂
Install the ODSC app, search for DeepOps on Tuesday and hit the ➕button to add my talk to your schedule. Check out MissingLink if you’re looking for a simple solution to your AI infrastructure problem.
