I woke up this morning (somewhat unsurprisingly). After poking my head out of the covers and feeling a cold chill in the air, I made the executive decision not to proceed with this course of action and promptly withdrew back under the covers. Ten minutes later, my alarm decides it is time to snooze no more and kindly invites me to follow suit. Begrudgingly I got up; another winter morning plagued by the Cold Start Problem.
In the same way owners look like their dogs, Machine Learning Scientists think like their AI. At Tractable, our AI for accident and disaster recovery suffers from the Cold Start Problem too, albeit all year round. Imagine you have an AI that is, say, really good at spotting scratches on Toyota Priuses, because for some reason you chanced upon a large, labelled dataset of Toyota Priuses that may or may not be damaged. Once the AI has trained by looking at all these images, it becomes a pro at knowing when a Prius is damaged and when it’s not.
Now, imagine you want to scale your AI to detect scratches on Lamborghini Aventadors instead. But you don’t have a large, labelled dataset of Lamborghini Aventadors. Your AI can’t really make an accurate judgement, simply because it doesn’t know what a Lamborghini Aventador is meant to look like, scratched or not. The AI, like the researcher who trained it, has trouble getting going when it’s cold, and needs to warm up to function smoothly.
Tractable’s AI is in use for visual car damage appraisal in thirteen countries over three continents. Every time we enter a new country,
- the cars look different and
- the way those cars are repaired is also different.
When we enter a new country, with new car types and repair standards, we often start cold. To get going, we start with our universal AI models and customize them as we get more and more data. But thanks to (1) and (2), this presents a whole set of challenges! It’s like trying to warm up your room when your furnace is broken!
In Machine Learning, (1) and (2) are types of domain shifts and they make the cold start problem even more challenging:
- Input shift: This happens when the inputs —in our case, images— that our AI model sees look different. There’s a wide spectrum covering what we would call a car and they might look very different.
- Conditional shift: This happens because the repair decisions, which are conditional on the damage to the car, are different around the world. To put that another way – if you have the exact same accident in Thailand, the UK and Poland, the car repairs will be different in each country. Our AI model can’t make the right decision unless it knows about these specifics.
- Output shift: The happens when you use the outputs of one domain for another. For example, in our case, the distribution of our labels—repair or don’t repair—would be very different for different countries. An AI model trained on Japanese data might be inclined to repair everything just because that’s what it has mainly seen from the data, even when used in other countries, where the methodology is very different.
When moving an AI model from one country to another, it needs to adapt to all these three types of shifts. Similar to how a Polish driver can’t drive easily in Japan and needs to adapt to the new environment and rules to drive safely, we need to adapt our AI when moving from one country to another! So the next question is how can we adapt? Can our AI trained for Japanese Adjusters write a repair estimate in Poland?
Tractable is working with the R&D Team at Georgian to expand our ability to handle domain shift. Our AI is data-hungry, and collecting enough of the right quality of data to enter a new country would be very time-consuming and expensive, so it’s critical that we can make the most of our existing dataset of cars from around the world. To address the three types of domain shift we encountered, we used techniques from two different areas of research: Ensemble Learning and Domain Adaptation.
1. Ensemble learning is the AI equivalent of getting a group of experts in a room, and putting their heads together to try and solve a difficult problem.
Many of us make our important decisions in life based on the opinions of others around us simply because often a decision made by a group of individuals results in a better outcome than a decision made by any one of them alone! Similarly, if we combine a diverse set of AI models (often called weak learners) the resulting model is more stable and performs better. This is referred to as Ensemble Learning. Although there are several Ensemble Learning methods, the most common are:
- Bagging: In bagging, the weak learners are trained independently from each other in parallel and are then combined by following some kind of deterministic averaging process.
- Boosting: in boosting, the weak learners are trained sequentially in a very adaptative way—meaning a weak learner is trained to focus on the misclassified examples of the previous weak learners—and then they are combined using a deterministic strategy.
- Stacking: Similar to bagging, the weak learners are trained in parallel and independently from each other, but unlike the previous two methods, the weak learners are combined by training a meta-model on top of them instead of using a deterministic strategy.
2. Domain Adaptation refers to a whole host of techniques designed to understand the similarities and differences between domains, so that we can control how much our AI can generalize or specialize.
As described previously, a domain shift is a change in the data distribution between an algorithm’s training data, and the data it encounters when deployed. Machine learning algorithms often adapt poorly to domain shifts. In Domain Adaptation, two common approaches are Instance-based and parameter-based methods.
a. Instance-based methods address domain shifts by weighting the training samples.
b. Parameter-based methods address domain shifts by adjusting the parameters of the models. Parameter-based methods are the most popular methods for deep learning models.
Through a combination of ensemble learning and domain adaptation techniques, Tractable can leverage its proprietary datasets to the fullest and expand its AI models to new markets quickly and efficiently. Our global AI can achieve a 20%+ performance boost over non-adapted models. We’ve been able to achieve expert-level AI, every time, using 10x less data than training from scratch. Perhaps I’ll set my thermostat to switch on the heating before I wake up tomorrow. Starting warm seems to work well for AI, so maybe it’ll work well for me too.
Editor’s note: Azin and Franziska are speakers for ODSC Europe 2021. Check out their talk, “Overcoming the Cold Start Problem: How to Make New Tasks Tractable,” there!
About the Authors/ODSC Europe 2021 Speakers on the Cold Start Problem:
Azin Asgarian is currently an applied research scientist on Georgian’s R&D team where she works with Georgian’s portfolio companies to help adopt applied research techniques to overcome business challenges. Azin holds a Master of Science in Computer Science from the University of Toronto and a Bachelor of Computer Science from the University of Tehran. Prior to joining Georgian, Azin was a research assistant at the University of Toronto and University Health Network (UHN) where she was working on the intersection of Machine Learning, Transfer Learning, and Computer Vision.
Franziska Kirschner is the Research and Product Lead of Car Inspection at Tractable. Her team uses machine learning to automate car damage appraisal across a range of applications. Her research interests include domain adaptation, and multitask- and multi-instance learning. In a previous life, she did a PhD in Physics at the University of Oxford. In her spare time, she enjoys cooking and making bad puns.
