In this episode of the neveropen Data Show, I spoke with Adam Marcus, co-founder and CTO of B12, a startup focused on building human-in-the-loop intelligent applications. We talked about the open source platform Orchestra,for coordinating human-in-the-loop projects; the current wave of human-assisted AI applications; best practices for reviewing and scoring experts; and flash teams.
Here are some highlights from our conversation:
Learn faster. Dig deeper. See farther.
Orchestra: A platform for building human-assisted AI applications
I spent a total of three years doing web-scale structured data extraction. Toward the end of that period, I started speaking with Nitesh Banta, my co-founder at B12, and we said, ‘Hey, it’s really awesome that you can coordinate all of these experts all over the world and give them all of these human-assisted AIs to take a first pass at work so that a lot of the labor goes away and you can use humans where they’re uniquely positioned.’ But we really only managed to make a dent in data extraction and data entry. We thought that an interesting work model was emerging here, where you had human-assisted AIs and they were able to help experts do way more interesting knowledge work tasks. We’re interested, at B12, about pushing all of this work up the knowledge work stack. The first stage in this process is to build out the infrastructure to make this possible.
This is where Orchestra comes in. It’s completely open source, it’s available for anyone to use on GitHub and contribute to. What Orchestra does is basically serve as the infrastructure for building all sorts of human-in-the-loop and human-assisted AI applications. It essentially helps coordinate teams of experts who are working on really challenging workflows and pairs them up with all sorts of automation, custom-user interfaces, and tools to make them a lot more effective at their jobs.
The first product that we built on top of Orchestra is an intelligent website product: a client will come to us and say that they’d like to get their web presence set up. Orchestra will quickly recruit the best designer, the best client executive, the best copywriter onto a team and it will follow a predefined workflow. The client executive will be scheduled to interview the client. Once an interview is completed, a designer is then staffed onto the project automatically. Human-assisted AI, essentially an algorithmic design, is run so that we can take some of the client’s preferences and automatically generate a few initial passes at different websites for them, and then the designer is presented with those and gets to make the critical creative design decisions. Other folks are brought onto the project by Orchestra as needed. If we need a copywriter, if we need more expertise, then Orchestra can recruit the necessary staff. Essentially, Orchestra is a workflow management tool that brings together all sorts of experts, automates a lot of the really annoying project management functionality that you typically have to bring project managers onboard to do, and empowers the experts with all sorts of automation so they can focus on what they’re uniquely positioned to do.
Bots and data flow programming for human-in-the-loop projects
Your readers are probably really familiar with things like data flow and workflow programming systems, and systems like that. In Orchestra, you declaratively describe a workflow, where various steps are either completed by humans or machines. It’s Orchestra’s job at that point, when it’s time for a machine to jump in (and in our case its algorithmic design) to take a first pass at designing a website. It’s also Orchestra’s job to look at which steps in the workflow have been completed and when it should do things like staff a project, notice that the people executing the work are maybe falling off course on the project and that we need more active process management, bring in incentives, and so forth.
The way we’ve accomplished all of this project automation in Orchestra is through bots; the super popular topic right now. The way it works for us is that Orchestra is pretty tightly integrated with Slack. At this point, probably everyone has used Slack for communicating with some kind of organization. Whenever an expert is brought into a project that Orchestra is working on, it will invite that expert to a Slack channel, where all of the other experts on his or her team are as well. Since the experts on our platform are using Orchestra and Slack together, we’ve created these bots that help automate process and project automation. All sorts of things like staffing, process management, incentives, and review hierarchies are managed through conversation.
I’ll give you an example in the world of staffing. Before we added staffing functionality to Orchestra, whenever we wanted to bring a designer onto a project, we’d have to send a bunch of messages over Slack, ‘Hey, is anyone available to work on a project?’ The designers didn’t have a lot of context, so sometimes it would take about an hour of work for us to actually do the recruiting, and experts wouldn’t get back to us for a day or two. We built a staffbot into Orchestra in response to this problem, and now the staffbot has a sense of how well experts have completed various tasks in the past, how much they already have on their plates, and the staffbot can create a ranking of the experts on the platform and reach out to the ones who are the best matches.
…Orchestra reaches out to the best expert matches over Slack and sends a message along the lines of, ‘Hey, here’s a client brief for this particular project. Would you like to accept the task and join the team?’ An expert who is interested just has to click a button, and then he or she is integrated into the Orchestra project and folded into the Slack group that’s completing that task. We’ve reduced the time to staff a project from a few days down to a little less than five minutes.
Related resources:
- Crowdsourcing at GoDaddy: How I Learned to Stop Worrying and Love the Crowd (a presentation by Adam Marcus)
- Why data preparation frameworks rely on human-in-the-loop systems
- Building a business that combines human experts and data science
- Metadata services can lead to performance and organizational improvements
