GRILLBot: A flexible conversational agent for solving complex real-world tasks

We present GRILLBot, a multi-modal task-oriented voice assistant to guide users through complex real-world tasks for the Alexa TaskBot Challenge. An effective TaskBot has to guide a user through a long and complex task, be engaging, and help solve problems along the way. GRILLBot achieves this in the domains of cooking and home improvement by helping search over a large task corpus with mixed- initiative and executing those tasks with neural dialogue management, knowledge grounded question answering, and web information extraction. To represent each task, we propose TaskGraphs as a dynamic graph unifying steps, requirements, and curated domain knowledge enabling detailed contextual explanations and adaptable task execution. Automatic linking of multi-modal elements helps the user navigate through the task and enrich the experience with helpful videos and images. Broad use of neural language models makes for flexible chit-chat, contextual intent parsing, and accurate task retrieval. Upon winning the competition, GRILLBot achieves a average rating in May (1st to 11th) of 3.86/5.0 and excels in longer conversations.