Editor's note: the TaskBot Challenge teams have been selected, you can learn more about them here.
More information on TaskBot Challenge
If you're interested in learning more about the TaskBot Challenge, visit the TaskBot FAQ page on the Alexa Prize website.
Amazon today announced that it is launching a new Alexa Prize TaskBot Challenge, in which university teams will compete to develop agents that assist customers in completing tasks requiring multiple steps and decisions.
It is the first conversational AI challenge to incorporate multimodal (voice and vision) customer experiences.
The application period for the challenge begins on March 17, and extends to April 16, 2021.
The new challenge will be conducted in parallel with the existing Socialbot Grand Challenge 4, in which nine university teams are competing to create socialbots that can converse coherently and engagingly with humans for 20 minutes on a range of topics.
Amazon science panel discusses Alexa Prize, TaskBot challenges
At WSDM 2021, seven Amazon scientists gathered for a roundtable event where Amazon Scholar Eugene Agichtein talked about the Alexa Prize Socialbot Grand Challenge and introduced the newly announced Alexa Prize TaskBot Challenge. Watch the panel talk about the research challenges in voice services and more.
“Customers worldwide interact with Alexa billions of times each week,” said Prem Natarajan, Alexa AI vice president, Natural Understanding. “Those interactions are goal-directed, such as ‘Alexa, what’s the weather forecast for tomorrow?’ or ‘Alexa, did the Lakers win last night?’. But increasingly customers want to go beyond these exchanges, to more complex, multimodal, multi-step tasks. Just as the existing Alexa Prize Grand Challenge is focused on advancing digital assistants’ ability to conduct multi-turn, open domain conversations, this new challenge will focus on what’s required of digital assistants to competently complete multi-step tasks for customers.”
“This new Alexa Prize challenge represents a major step towards Alexa becoming the best digital assistant, by interactively assisting customers to complete everyday tasks, be it in cooking or home improvement,” said Yoelle Maarek, vice president of research and science, Alexa Shopping. “This is a hard AI challenge and we need to rally the best scientific minds if we want to be successful. I am delighted to see that our scientists and scholars at Amazon are turning once more to the academic community to jointly address it. This is a wonderful example of our customer-obsessed science approach where we push the boundaries of science to help and delight our customers together with academia.”
The goal of the new TaskBot Challenge is to help advance the science of conversational AI, but in ways that differentiate it from the existing Socialbot Challenge, says Eugene Agichtein, a computer science professor at Emory University, and an Amazon Scholar. Agichtein, who joined Amazon as a scholar in 2019, is very familiar with the Alexa Prize competition; he was the faculty advisor for Emory’s Alexa Prize team the first two years of the competition. The team from Emory won the most recent Alexa Prize socialbot challenge.
“The goal of the socialbot challenge is ambitious and exciting from a scientific perspective,” Agichtein said. “But the focus hasn’t been on how helpful the socialbot can be in actually assisting people. We wanted to design a new challenge that was not only interesting from a science perspective, but also helps customers complete tasks, or solve problems.”
TaskBot Challenge
The idea for the new challenge emerged last year, and aligns with a goal for Alexa to create next-generation conversational AI shopping experiences by engaging customers in pre- and post-purchase dialogues. The TaskBot Challenge will run for three years, and initially teams will focus on two domains: cooking and home improvement. The challenge incorporates multimodal customer experiences, so in addition to receiving verbal instructions, customers with Echo screen devices, such as the new Echo Show 10, could also be presented with step-by-step instructions, images, or diagrams that enhance task guidance.
For example, a customer might ask Alexa how to fix a scratch on a car. The TaskBot will then ask the customer more questions about their task, and then interactively provide step-by-step instructions and explanations for each step, or potentially adjust its plan based on customer input.
After the interaction ends, the customer will be asked to rate how helpful that TaskBot was with the task, and will have the option to provide freeform feedback to help the teams improve their TaskBot.
Success in the challenge will require participants to advance the state of the art in conversational AI, and address difficult science challenges related to knowledge representation and inference, commonsense and causal reasoning, and language understanding and generation, among others — requiring synthesis of multiple areas and approaches in AI.
“In developing the TaskBot Challenge, we tried to set a goal that is scientifically ambitious and novel, yet potentially achievable within a three-year time horizon,” Agichtein explained. “For example, the participants will have to integrate into the interaction the domain knowledge from structured and unstructured sources, such as databases of recipes and ingredients, with commonsense and causal reasoning to understand if a step in a recipe is not possible. Interacting with millions of customers attempting to accomplish tasks in the messy real world will be humbling, challenging, and yet inspiring experience for university students.”
Interacting with millions of customers attempting to accomplish tasks in the messy real world will be humbling, challenging, and yet inspiring experience for university students.
Another scientific challenge will be how the participating teams guide a customer through complex, multi-step plans that may need to be revised if, for instance, the customer needs to substitute an ingredient, or doesn’t have a tool required to complete the task.
“That’s where things get really challenging” Agichtein said. “The TaskBot must first develop a plan — baking a cake, for instance — and then lead the customer through the baking process. The TaskBots will have to understand when customers are getting into trouble, say, if they have run out of flour. The TaskBots will then have to suggest solutions to such problems and adjust the plan as necessary.”
In year one of the competition, Agichtein expects teams to focus primarily on single-session tasks, but teams have to be prepared to maintain and resume tasks over multiple sessions, perhaps extending across multiple days.
“In year one, we won’t expect the TaskBots to successfully handle very complex tasks, especially those that span multiple sessions, but it’s a goal we’ll want teams to eventually address over the course of the challenge,” he said.
Other challenges the teams will confront is what tasks to try to help with, and what tasks are inappropriate or dangerous, and have to be declined.
The deadline for university teams to apply for the challenge is April 16, 2021. Up to ten teams will be selected to participate in the challenge by June 11, and the competition will begin on June 14. The year-long competition will conclude in May 2022, with winners being announced the following month.
Teams selected for the challenge receive a $250,000 research grant, Alexa-enabled devices, free Amazon Web Services (AWS) cloud computing services to support their research and development efforts, access to the TaskBot Toolkit, as well as other resources data, and Alexa team support. The winning team receives a $500,000 prize, and the second- and third-place teams receive prizes of $100,000 and $50,000, respectively.
Alexa Prize Socialbot Grand Challenge
The Alexa Prize first launched in 2016 as a competition for university students dedicated to advancing the field of conversational AI. Teams are challenged to design socialbots that Alexa customers can interact with via Alexa-enabled devices. The student teams’ ultimate goal is to meet the Grand Challenge: earn a composite score of 4.0 or higher (out of 5) from the judges, and have the judges find that at least two-thirds of their conversations with the socialbot in the final round of judging remain coherent and engaging for 20 minutes.
It’s now the quarter-finals of the @AmazonScience Alexa Prize Socialbot Grand Challenge 4. Say “Alexa, let’s chat”—to a device or phone app—and you’ll be connected to the socialbot of one of the university teams, and maybe it’ll be @Stanford’s Chirpy Cardinal #AlexaPrize #NLProc pic.twitter.com/JYNBzKrJ09
— Stanford NLP Group (@stanfordnlp) March 8, 2021
The teams selected for the challenge receive a $250,000 research grant, Alexa-enabled devices, free Amazon Web Services (AWS) cloud computing services to support their research and development efforts, access to the Cobot (conversational bot) toolkit and other tools, data, and Alexa team support.
In previous challenges, participating teams have improved the state of the art for open domain dialogue systems by developing improved natural language understanding (NLU) systems, neural response generation models, common sense knowledge modeling, and dialogue policies leading to smoother, and more engaging conversations. Alexa Prize also has led to innovative solutions that are now incorporated into existing customer experiences, such as an explicit content filter and neural response generator.
A team from the University of Washington won the inaugural competition. In 2018, a team from the University of California, Davis won the challenge, and the team from Emory University won last year. Research papers are published each year by the participating teams, and by the Amazon Alexa Prize team. The papers are accessible from the Alexa Prize website.
Nine university teams from around the globe are currently participating in Alexa Prize Socialbot Grand Challenge 4. The challenge began last November and will conclude in August 2021. The winning team receives a $500,000 prize, and the second- and third-place team receive prizes of $100,000 and $50,000, respectively. The grand challenge, a $1 million research grant, will be awarded to the winning team’s university if it attains a composite score of 4.0 or higher, on a 5-point scale, and at least two-thirds of their socialbot’s conversations with interactors last for 20 minutes.
Customers can engage with one of the existing competitions’ socialbots simply by saying, “Alexa, let’s chat".
