SlugJARVIS: Multimodal commonsense knowledge-based embodied AI for SimBot Challenge

We present SlugJARVIS, an advanced multimodal, task-oriented embodied AI developed for the purpose of assisting users in navigating complex real-world tasks within the context of the Alexa Prize SimBot Challenge. SlugJARVIS is designed to effectively maintain user engagement, accurately interpret user intent, make informed and executable decisions based on the current environment, and adeptly manage unforeseen situations. This functionality is achieved through a comprehensive system that encompasses single- and multimodal understanding, systematic task management, and goal-oriented response generation.