Alexa Prize Proceedings

We describe our work to date on using large language models (LLMs) in our Alexa Prize Socialbot. Our work has laid the groundwork for looking more closely at using LLMs in a conversational system. We also analyze common patterns in conversations our bot has had with users.

In this technical report, we present EMMA, a foundation model for embodied, interactive, and multimodal task completion in 3D environments. Different to previous Vision+Language (V+L) models, EMMA is an encoder-decoder architecture that encodes both images and videos (i.e., sequences of frames), and it is able to generate natural language tokens conditioned on specific task prompts. By treating every task

We present Symbiote, an embodied home assistant that maps images from its camera into objects and rooms, builds geometric semantic maps, parses human instructions and conversations into user intents and their arguments, explores in a goal-directed way to find relevant objects (if not present in the map) and executes the inferred actions plans using its navigation and manipulation policies, and/or ask questions

We present SlugJARVIS, an advanced multimodal, task-oriented embodied AI developed for the purpose of assisting users in navigating complex real-world tasks within the context of the Alexa Prize SimBot Challenge. SlugJARVIS is designed to effectively maintain user engagement, accurately interpret user intent, make informed and executable decisions based on the current environment, and adeptly manage unforeseen

The growing demand for advanced AI necessitates the development of an intelligent agent capable of perceiving, reasoning, acting, and communicating within an embodied environment. We introduce SEAGULL, an interactive embodied agent designed for the inaugural Alexa Prize SimBot Challenge, which can complete complex tasks in the Arena simulation environment through dialog with users. SEAGULL is engineered

Embodied Artificial Intelligence is an emerging field that focuses on creating intelligent agents that can perceive, navigate, and manipulate objects in their environment. While current smart assistants are limited to speech and text-based interactions, developing embodied agents that can engage in natural dialogue and complete physical tasks presents a significant challenge—but one necessary to the advancement

This paper introduces SalsaBot, an embodied agent designed for the first Alexa Prize Simbot Challenge. The primary aim of SalsaBot is to assist users in completing a game within a virtual environment by providing a consistent user-centric experience, which requires the agent to be capable of handling various types of user interactions. Individuals from various backgrounds interact with SalsaBot using Automatic

As artificial intelligence continues to advance, the development of intelligent assistive agents that can help people in completing daily tasks has become an area of increasing interest. However, the automation of such agents presents several challenges: (1) the enormous variety of user-defined tasks, (2) the ability of complex reasoning and visual comprehension for carrying out actions, and (3) natural

We introduce our system BEAST (Building an Embodied Action-prediction System with Trajectory data) for interactive instruction-following within the Alexa Arena Platform. Our system leverages the abstraction of navigation provided by the Arena to decouple the language and vision predictions. This allows for greater simplicity within the system, and for rapid augmentation of the trajectory data-set and training

We introduce an embodied AI agent designed to interact with human users, achieve high mission success rates, and deliver a satisfying user experience. Our agent utilizes a modular framework, incorporating explicit state updates based on previous actions, next-action predictions derived from user input, target object detection from observations, and various exploration strategies for locating unseen targets