Sense, Act, and Scale
The path to improving building energy efficiency can be paved with the framework of sense, act, and scale say authors Bharathan Balaji, an Amazon senior research scientist within the company's Devices organization, and Rob Aldrich, an Amazon Web Services senior sustainability strategist.

Creating sustainable, data-driven buildings

As office buildings become smarter, it is easier to configure them with sustainability management in mind.

Editor’s note: This article is adapted from a keynote presentation Bharathan Balaji , an Amazon senior research scientist within the company’s Devices organization, delivered in June at the 17th International Conference on Intelligent Environments . It is further informed by the book "IP-Enabled Energy Management, A Proven Strategy for Administering Energy as a Service " and its author, Rob Aldrich, Amazon Web Services senior sustainability strategist.

Buildings generate about 28% of the global greenhouse gas emissions today. The United Nations Global Status Report projects that buildings need to be at least 30% more energy efficient to achieve Paris Agreement goals.

How can we achieve that 30% energy efficiency target?

The path to reducing our emissions by improving building energy efficiency can be paved with the framework of sense, act, and scale. We need to sense to ascertain efficiency gaps within buildings. We need solutions that act on the information to achieve energy savings. And finally, we need to scale solutions so they get implemented broadly. Here is how this proposed framework can help us achieve our goals.

Sense

For office buildings that are smart and connected, the data set is rich and has much of the granular, sustainability data needed to drive change. Electricity and gas meters tell us how much energy is being consumed by a building, occupancy sensors tell us the number of people in the building, and temperature sensors tell us how much energy we need to cool a room. Sensors are the source of our information and the key to unlocking energy efficiency gaps. Even simple dashboards with such data can motivate users to save energy.

These types of sensors are abundant in modern buildings. However, many of them are wired sensors that are part of the building’s original design, and it is expensive to modify or install new sensors. Office buildings have a life of 50+ years, and sensor technology advances far more rapidly. Wireless sensors undoubtedly reduce communications costs, but they still need to be powered through wires, or use batteries that significantly increase maintenance costs at scale (imagine changing the batteries in every room of an office building).

New sensor options provide for ambient energy harvesting. These wireless sensors work by scavenging energy from the environment such as using ambient light, ventilation air flow, or hot water pipes. These sensors can minimize both energy and communications costs, but scavenged energy is insufficient to sense 24x7. We can improve reliability by predicting the environmental patterns and judiciously using the available energy.

A recent paper in SenSys (coauthored by Bharathan, lead author of this article) showed that reinforcement-learning-based scheduling of energy harvesting sensors can detect 93% of events in a real-world deployment. While the small percentage of missed events make these sensors ineligible for use in essential services, we can use the data from these inexpensive sensors opportunistically to create a rich information layer that helps save energy.

Information Bottleneck: Senors

This new, rich information layer can drive the return on investment (ROI) that has been lacking in many sensor installations. Energy and data managers can provide the missing link between top-end sustainability initiatives and the many different sensor options that exist in buildings. Furthermore, the cost of sensor architectures can be reduced by focusing only on the key data sources that support a given use case. 

For this article we chose to focus primarily on building sustainability data: energy, occupancy, emissions, air and water. This focus helps enable an estimated ROI because you already have a use case that defines how you will act on the information available. The use case for sustainability is to reduce wasted energy while moving to low greenhouse gas (GhG) fuel sources.  Informed by sensor data, the actions taken in support of these goals can be the mechanism by which savings are achieved.

Act

The traditional way to make buildings more energy efficient is to inspect the equipment, install sensors to measure baseline energy consumption, fix faults, upgrade equipment, and optimize equipment configuration. Heating, ventilation and air conditioning (HVAC) systems typically comprise the largest portion of building energy use, and many of the efficiency measures target HVAC improvements. These methods work, and can lead to more than 10% reductions in building energy use. The entire process is often referred to as building retrofitting through performance contracting.

However, two issues with the above approach typically block adoption. First, there is an upfront cost to hire experts and upgrade equipment. The ROI can take years. Second, there is limited scope for innovation beyond the template followed during commissioning. Building innovation is stifled by vertically integrated systems and an inability to easily deploy third-party applications. One of the primary reasons for the explosive growth in the computing industry is a standard interface and ease of application installation. An analogous system for buildings will create new opportunities to save energy. The innovation opportunities with a standardized building information system is highlighted with three use cases below. It is easy to create such a system with current technologies; the figure below shows a high-level architecture.

Building information system architecture

Occupancy-based control

The idea is simple: if we shut off systems that aren’t required when people aren’t present, we save energy. However, detecting occupancy reliably in a privacy-preserving manner is challenging, and most buildings today keep the lights (and HVAC) on even when no one is present. A paper published in SenSys (coauthored by Bharathan) showed that it is possible to infer occupancy using WiFi data, building floor plans, and personnel office room assignments. Among the study participants, peak building occupancy was just 60% (see figure below), and occupancy-based control saved 18% of HVAC electricity use by controlling one-quarter of the building area. The proposed solution simply leverages existing building infrastructure and is inexpensive to deploy. This type of solution is possible only because the information across different systems is exchanged freely.

Building Occupancy Trends

Fault detection

Fixing faults is core to building maintenance, but it is challenging to identify energy-wasting faults as they are difficult to notice, unlike a leak or an uncomfortable temperature. Typical building-fault detection relies on protocols established by experts, but these rules do not provide sufficient prioritization information, nor how much energy they waste.

Sophisticated fault detection algorithms have been published in literature, yet these are not deployed in practice because of vendor- locked systems. Using one year of building data, researchers (Bharathan was a coauthor) developed a simple machine learning algorithm that looks for rooms that do not follow typical temperature patterns. The algorithm identified 88 faults within the building’s HVAC system after an expert fixed all the faults found during an inspection. Many of these faults had existed for years, and resulted in estimated 410.3MWh/year savings. Again, the key component to this solution: easy access to building data.

Software thermostat

The thermostat is the only interface between building occupants and the energy-intensive HVAC system. And yet, in most buildings, occupants don’t know where the thermostat is or how to use it. The HVAC system’s primary function is to keep occupants comfortable so that they can be productive. But without thermostat feedback, occupants can end up being uncomfortable and waste energy.

With the building information system, researchers (Bharathan and collaborators) built a software version of the thermostat to address these concerns (screenshot below). The application was an instant hit and remains popular eight years after its launch. The resulting user study published in Ubicomp showed that users were frustrated with the old thermostat. In fact, one user actually taped a manila envelope on the vent to stop cold air from blowing. The software thermostat helped users precisely control their environment and send complaints if needed. The HVAC maintenance personnel were worried that the interface would lead to a flood of complaints that they weren’t staffed to handle. Usage data showed that most users were happy to use the application without giving any feedback. The few complaints received led to identification of major faults, such as a thermostat being blocked by a computer.

Software thermostat

The three use cases above didn’t require additional sensor installations and simply leveraged existing information. With low-cost solutions, we can attract building owners to adopt solutions that save energy. But we need additional incentives within the building industry to create these low-cost solutions that can have large-scale impact.

These use cases demonstrate that sustainable design doesn’t stop at the brick and mortar of the building. It should carry through to how the energy, emissions, air, water and waste can be managed as systems across buildings. As companies worldwide embark on making their buildings more sustainable, it will be critical to have a data-driven measure of success. The sense and act steps allow each company to look at what is common in the data model today, get started, assess the value, and scale as needed using open-source tools.

Scale

Even when an attractive energy-saving solution is available, it is difficult to deploy the solution at scale. This is because each building is unique, from its infrastructure and how it is used, to the software used to manage daily operations, and how it changes over time. While the fundamental components of a building remain the same (e.g., rooms, smoke sensors, ventilation fans), each vendor treats them differently. When we try to deploy a solution to a building, the discrepancies between vendor representations become difficult to resolve automatically.

In the computing industry, on other hand, it is easy for us to install an application without worrying about the manufacturer or provider because of the use of specifications (e.g., standard protocols for WiFi) and programming interfaces (e.g., Android OS for the phone). Researchers (including Bharathan) created such a standard interface for buildings with the Brick schema, where the building components and their connections to each other are represented through a knowledge graph. The figure below shows a Brick representation of a toy building with two rooms and a few sensors. Brick is now an industry consortium with growing demand, and is in the process of being integrated into building standards.

Given a standard representation such as Brick, we still have the task of representing the existing building in this new format, which can take manual effort and be slow to deploy. Using machine-learning techniques in natural language processing, we can automate this translation and minimize manual effort. The algorithm’s performance improves as more buildings are mapped to Brick and it learns from representation patterns across buildings.

The Brick schema

With the sense, act and scale framework, we envision a day when it will be as easy to configure a building as it is our phones today. We can improve the information available to building managers by using low-cost sensors, use the available information to develop innovations that save energy, and deploy the solution to many buildings with use of a knowledge graph.

Getting started

We are seeing early success in using the sense, act, scale approach in our AWS Sustainability Services practice to optimize how buildings report their sustainability data through the cloud.  It solves several problems by providing a simple framework to plan how our top-level sustainability strategy can be supported by specific building-optimization steps, underpinned by a semi-standardized data model.

The lack of standardization across building management systems has resulted in difficulties in accessing the data. Now that those data acquisition problems are being solved through advances in IoT and API, it opens up new opportunities to expose, analyze and report data that was previously difficult or costly to acquire.  With new advances like the Brick schema, we are making advances in how we can manage building assets at scale, just like servers, laptops and phones.

We are starting to see the potential to move the world from a building management systems approach; one building, one manager to a building systems management approach; many buildings, one manager. Energy efficiency gains of 30% or more are more feasible when we automate energy-control policies across all buildings at the push of a button.

Research areas

Related content

US, WA, Seattle
The Annapurna ML team is looking for a Senior Applied Scientist to work on the intersection of Artificial Intelligence and program analysis to raise the code quality bar in our state-of-the-art deep learning compiler stack. This stack is designed to optimize application models across diverse domains, including Large Language and Vision, originating from leading frameworks such as PyTorch, TensorFlow, and JAX. Your role will involve working closely with our custom-built Machine Learning accelerators, Inferentia and Trainium, which represent the forefront of Annapurna innovation for advanced ML capabilities, and is the underpinning of Generative AI. As a Senior Applied Scientist, you'll be instrumental in designing, developing, and deploying analyzers for ML compiler stages and compiler IRs. You will architect and implement business-critical tooling, publish research, and mentor a brilliant team of experienced scientists and engineers. You will need to be technically capable, credible, and curious in your own right as a trusted scientist, innovating on behalf of our customers. Your responsibilities will involve tackling crucial challenges alongside a talented engineering team, contributing to leading-edge design and research in compiler technology and deep-learning systems software. Strong experience in programming languages, compilers, program analyzers, and program synthesis engines will be a benefit in this role. A background in machine learning and AI accelerators is preferred but not required. A day in the life Diverse Experiences Amazon values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why Amazon? We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at Amazon, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud.
US, WA, Seattle
Join the AWS Perimeter Protection team as a Senior Applied Scientist, where you will bring your deep ML engineering expertise to design, build, and scale AI-driven security solutions that protect AWS customers worldwide. This role is ideal for someone who has already built and shipped production ML systems at industry scale and is looking to apply that experience to high-impact security challenges. You will own the full ML lifecycle — from research and prototyping to production deployment and optimization — powering services including Web Application Firewall, DDoS Protection, Bot Management, and Infrastructure Protection. With services spanning all AWS regions and handling trillions of requests per week, you will solve complex engineering and science problems where model performance, system reliability, and low-latency inference are critical. Key job responsibilities - Design, build, and deploy production-grade ML models and systems for real-time threat detection, mitigation, and protection against evolving cyber threats at cloud scale. - Own the full ML lifecycle end-to-end — from problem formulation, data engineering, and model development through to production deployment, monitoring, and continuous improvement. - Architect and optimize ML pipelines, training infrastructure, and serving systems to meet strict latency, throughput, and reliability requirements at AWS scale. - Bridge the gap between research and production by translating novel ML approaches into robust, scalable, and maintainable systems that operate in real-time security environments. - Design and implement feature engineering workflows and large-scale data processing pipelines to support rapid experimentation and reliable model iteration. - Collaborate closely with software engineering teams to integrate ML models into distributed, low-latency security services, driving engineering decisions around model serving, infrastructure, and system design. - Analyze large-scale production data to identify patterns, anomalies, and emerging threat vectors, and translate findings into measurable improvements to detection and mitigation capabilities. - Establish and improve best practices for ML system design, model evaluation, A/B testing, and production monitoring across the team. - Mentor junior scientists and engineers, raising the bar on both scientific rigor and engineering quality.
US, WA, Seattle
The Automated Reasoning Group in the Amazon Neuron team is looking for an Applied Scientist to work on the intersection of Artificial Intelligence and program analysis to raise the code quality bar in our state-of-the-art deep learning compiler stack. This stack is designed to optimize application models across diverse domains, including Large Language and Vision, originating from leading frameworks such as PyTorch and JAX. Your role will involve working closely with our custom-built Machine Learning accelerator, Trainium, which represents the forefront of innovation for advanced ML capabilities, and is the underpinning of Generative AI. In this role as an Applied Scientist, you'll be instrumental in designing, developing, and deploying analyzers for ML compiler stages and compiler IRs. You will architect and implement business-critical tooling, publish research, and mentor a brilliant team of experienced scientists and engineers. You will need to be technically capable, credible, and curious in your own right as a trusted AWS Neuron engineer, innovating on behalf of our customers. Your responsibilities will involve tackling crucial challenges alongside a talented engineering team, contributing to leading-edge design and research in compiler technology and deep-learning systems software. Strong experience in programming languages, compilers, program analyzers, theorem provers, and program synthesis engines will be a benefit in this role. A background in machine learning and AI accelerators is preferred but not required.
IN, KA, Bengaluru
Customer addresses, Geospatial information and Road-network play a crucial role in Amazon Logistics' Delivery Planning systems. We own exciting science problems in the areas of Address Normalization, Geocode learning, Maps learning, Time estimations including route-time, delivery-time, transit-time predictions which are key inputs in delivery planning. As part of the Science team within DNT (Last Mile), you will partner closely with other scientists and engineers in a collegial environment to develop enterprise ML solutions with a clear path to business impact. The setting also gives you an opportunity to think about a complex large-scale problem for multiple years and building increasingly sophisticated solutions year over year. In the process there will be opportunity to innovate, explore SOTA and publish the research in internal and external ML conferences. Successful candidates will have deep knowledge of competing machine learning methods for large scale predictive modelling, natural language processing, semi-supervised & graph based learning. We also look for the experience to graduate prototype models to production and the communication skills to explain complex technical approaches to the stakeholders of varied technical expertise. Here is a glimpse of the problem spaces and technologies that we deal with on a regular basis: 1. De-duping and organizing addresses into hierarchy while handling noisy, inconsistent, localized and multi-lingual user inputs. We do this at the scale of millions of customers for established (US, EU) as well as emerging geographies (IN, MX). We make use of technologies like LLMs, Weak supervision, Graph-based clustering & Entity matching. We also use additional modalities such as building outlines in maps, street view images and 3P datasets, gazetteers. 2. Build a generic ML framework which leverages relationship between places to improve delivery experience by learning precise delivery locations and propagating attributes, such as business hours and safe places. 3. (Work done in sister teams) Developing systems to consume inputs from areal imagery and optimize our maps to enable efficient delivery planning. Also building models to estimate travel time, turn costs, optimal route and defect propensities. Key job responsibilities As an Applied Scientist I, your responsibility will be to deliver on a well defined but complex business problem, explore SOTA technologies including GenAI and customize the large models as suitable for the application. Your job will be to work on a end-to-end business problem from design to experimentation and implementation. There is also an opportunity to work on open ended ML directions within the space and publish the work in prestigious ML conferences. About the team Last Mile Address Intelligence (LMAI) team owns WW charter for address and location learning solutions which are crucial for efficient Last Mile delivery planning. The team works out of Hyderabad and Bangalore offices in India. LMAI is a part of the DNT Science team, which also owns problems in the space of maps learning and travel time estimations. The rest of the DNT Science team and senior leadership of Last Mile org works out of Bellevue office.
US, CA, Pasadena
We are seeking a Research Science Manager to join the SAF Lab. In this role, you will lead a team of world-class applied scientists, engineers, post-docs and interns developing the next generation of safe autonomy on highly dynamic robotic systems. You will drive technical vision, research strategy, and ensure your team's innovations translate into production systems that operate at Amazon scale. You will interface with top academic researchers at the forefront of safe autonomy, and interdisciplinary teams across Amazon working on autonomous mobile robots, mobile manipulators, and dynamically stable robots. You will bridge academic research with real-world deployable safety layers that enable robots to safely operate around humans. Key job responsibilities • Manage a team of scientists and engineers developing a universal safety layer for robotic systems, with a focus on the next generation of robots • Work with leadership to design and execute multi-year research roadmaps for the development of safe autonomy that spans all types of current and emerging robotic platforms • Lead Scientists, Engineers, post-docs and interns to realize research and development goals and provide evidence of these results in a variety of formats • Drive integration of control barrier functions with planning, perception and learning while maintaining guarantees of safe high-performance robot behavior • Collaborate with product teams and science leaders to set a science roadmap (with eventual impact on real robots). • Publish research findings at top-tier conferences and contribute to the broader robotics community • Build relationships with academic and industry partners to stay at the forefront of safe autonomy A day in the life Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: 1. Medical, Dental, and Vision Coverage 2. Maternity and Parental Leave Options 3. Paid Time Off (PTO) 4. 401(k) Plan If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply! About the team Work with the inventor of control barrier functions in the Safe Autonomy Frontiers (SAF) Lab. The first industry research lab in safe autonomy, developing a universal safety layer for the next generation of robotic systems: mobile robots, manipulators, mobile manipulators, and future platforms with dynamic stability. You will push the frontiers of performant safety for highly dynamic robots: CBF theory integrated with perception and learning, evaluated on next-generation robots. Your work will underpin robots operating alongside people at Amazon's unprecedented scale.
US, WA, Bellevue
Transform global logistics by developing innovative technological solutions that optimize complex international shipping networks. Join a dynamic team where your advanced scientific expertise will directly impact millions of packages moving across borders, solving intricate transportation challenges through applied science. Key job responsibilities - Develop advanced deep learning and generative AI applications for sophisticated product classification systems - Lead complex modeling analyses to support strategic business decision-making and policy development - Partner with engineering and operations teams to design transformative machine learning solutions for transportation challenges - Design and document technical strategies and applied science roadmaps - Innovate and provide technological direction in international imports and exports logistics A day in the life Your workday will be an exciting blend of data exploration, innovative problem-solving, and collaborative technology development. You'll dive into complex datasets, design sophisticated machine learning models, engage with cross-functional teams, and create technological solutions that reshape international shipping capabilities. About the team We are a passionate group of scientists and engineers dedicated to building comprehensive transportation technologies. Our mission is to create highly scalable, intelligent systems that enable seamless international package movement. By pushing technological boundaries, we're reimagining global logistics through innovative applied science. Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: 1. Medical, Dental, and Vision Coverage 2. Maternity and Parental Leave Options 3. Paid Time Off (PTO) 4. 401(k) Plan Learn more about our benefits here: https://www.amazon.jobs/en/internal/benefits/us-benefits-and-stock
US, WA, Seattle
Innovators wanted! Are you an entrepreneur? A builder? A dreamer? This role is part of an Amazon Special Projects team that takes the company’s Think Big leadership principle to the limits. If you’re interested in innovating at scale to address big challenges in the world, this is the team for you. As an Applied Scientist on our team, you will focus on building state-of-the-art ML models for healthcare. Our team rewards curiosity while maintaining a laser-focus in bringing products to market. Competitive candidates are responsive, flexible, and able to succeed within an open, collaborative, entrepreneurial, startup-like environment. At the forefront of both academic and applied research in this product area, you have the opportunity to work together with a diverse and talented team of scientists, engineers, and product managers and collaborate with other teams. This role offers a unique opportunity to work on projects that could fundamentally transform healthcare outcomes. Key job responsibilities In this role, you will: • Design and implement novel AI/ML solutions for complex healthcare challenges • Drive advancements in machine learning and data science • Balance theoretical knowledge with practical implementation • Work closely with customers and partners to understand their requirements • Navigate ambiguity and create clarity in early-stage product development • Collaborate with cross-functional teams while fostering innovation in a collaborative work environment to deliver impactful solutions • Establish best practices for ML experimentation, evaluation, development and deployment • Partner with leadership to define roadmap and strategic initiatives You’ll need a strong background in AI/ML, proven leadership skills, and the ability to translate complex concepts into actionable plans. You’ll also need to effectively translate research findings into practical solutions. A day in the life You will solve real-world problems by getting and analyzing large amounts of data, generate insights and opportunities, design simulations and experiments, and develop statistical and ML models. The team is driven by business needs, which requires collaboration with other Scientists, Engineers, and Product Managers across the Special Projects organization. You will prepare written and verbal presentations to share insights to audiences of varying levels of technical sophistication. About the team We represent Amazon's ambitious vision to solve the world's most pressing challenges. We are exploring new approaches to enhance research practices in the healthcare space, leveraging Amazon's scale and technological expertise. We operate with the agility of a startup while backed by Amazon's resources and operational excellence. We're looking for builders who are excited about working on ambitious, undefined problems and are comfortable with ambiguity.
US, TX, Austin
Amazon Leo is an initiative to launch a constellation of Low Earth Orbit satellites that will provide low-latency, high-speed broadband connectivity to unserved and underserved communities around the world. As a Systems Engineer, this role is primarily responsible for the design, development and integration of communication payload and customer terminal systems. The Role: Be part of the team defining the overall communication system and architecture of Amazon Leo’s broadband wireless network. This is a unique opportunity to innovate and define groundbreaking wireless technology at global scale. The team develops and designs the communication system for project Leo and analyzes its overall system level performance such as for overall throughput, latency, system availability, packet loss etc. This role in particular will be responsible for leading the effort in designing and developing advanced technology and solutions for communication system. This role will also be responsible developing advanced physical layer + protocol stacks systems as proof of concept and reference implementation to improve the performance and reliability of the LEO network. In particular this role will be responsible for using concepts from digital signal processing, information theory, wireless communications to develop novel solutions for achieving ultra-high performance LEO network. This role will also be part of a team and develop simulation tools with particular emphasis on modeling the physical layer aspects such as advanced receiver modeling and abstraction, interference cancellation techniques, FEC abstraction models etc. This role will also play a critical role in the integration and verification of various HW and SW sub-systems as a part of system integration and link bring-up and verification. Export Control Requirement: Due to applicable export control laws and regulations, candidates must be a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum.
US, CA, Sunnyvale
We are looking for a Senior Applied Scientist to help drive the research and development of real-time multimodal conversational AI. You will contribute across two focus areas: advancing foundation models for speech and audio, and building the post-training systems (reward modeling, reinforcement learning) that shape natural, human-like conversational behavior. You will own a significant research area and contribute across the full model lifecycle — from pre-training and architecture design through post-training alignment and real-time deployment. You will work at the frontier of what’s possible in conversational AI, with the compute, data, and runway to pursue problems that few teams in the world have the resources to tackle. As a Senior Scientist, you will drive the technical execution of your research area, contribute to the team’s roadmap, and work closely with inference engineers to ensure your models are designed for real-time production deployment. Key job responsibilities What You’ll Do Foundation Model Scaling - Help build and train large-scale multimodal foundation models for real-time speech and audio generation, from architecture design through production-scale training - Advance the scaling and efficiency of conversational models, including the relationship between data, model size, and real-time performance - Design model architectures informed by hardware constraints and inference requirements, working with inference engineers to ensure models are servable from inception - Develop training methodologies for multimodal models that jointly process and generate speech, language, and audio in real-time streaming contexts - Contribute to the state of the art on efficient architectures and training methods for conversational AI at scale Post-Training & Reinforcement Learning - Design and build reward models and reward functions for speech systems — capturing naturalness, fluency, conversational quality, and real-time responsiveness - Develop and apply reinforcement learning methods to shape conversational behavior — teaching models natural timing, responsiveness, and fluid interaction - Build parts of the post-training pipeline from SFT through RL alignment, optimized for real-time multimodal outputs rather than text-only generation - Design evaluation frameworks that capture the quality dimensions unique to real-time conversation (latency sensitivity, audio quality, prosody, interaction naturalness) Real-Time Perception & Generation - Advance the team’s capabilities in real-time perception — the ability of the model to process incoming audio/speech while simultaneously generating responses - Develop techniques for natural interactive systems where the model handles concurrent input and output with human-like timing - Work at the intersection of model architecture and production constraints to ensure multimodal capabilities function within hard real-time latency budgets
US, WA, Bellevue
Robots are about to step out of the movies and into our homes. If you're excited about this robotics revolution, come join Amazon's Fauna Robotics. As an Applied Scientist in this org, you'll push the state-of-the-art techniques in Robotics. You'll drive technical excellence in areas such as perception, manipulation, sim2real transfer, reinforcement learning and multi-task learning, designing novel algorithms that bridge the gap between research and real-world deployment. In this role, you will integrate hands-on technical expertise with scientific leadership, ensuring your team delivers robust solutions for dynamic real-world environments. While staying updated with the latest advancements in robotics research, you will leverage your experience and expertise to guide the team towards the most promising direction. Collaborating with a talented group of software engineers and scientists, you will bring your ideas to fruition. Deploying robots in home environments requires you to anticipate challenges not encountered in structured settings like warehouses. Additionally, consumer-grade sensors and actuators necessitate working around their limitations, requiring out-of-the-box innovative ideas. As such, you are someone who truly enjoys thinking about problems using first principles at a holistic and systems level to address the aforementioned challenges. Key job responsibilities - Lead technical initiatives in areas such as robotics foundation models, reinforcement learning and manipulation - - Design experiments to identify the limitations of current state-of-the-art models and developing new models or techniques that can surpass these models - Design and implement novel deep learning architectures that push the boundaries of what robots can understand and accomplish - Mentor fellow scientists while maintaining strong individual technical contributions - Collaborate with engineering teams to optimize and scale models for real-world applications - Influence technical decisions and implementation strategies within your area of focus - Experienced in communicating complex technical work to a non-technical audience - The ability to work with minimal guidance, be proactive and to handle ambiguity and the challenge of quickly evolving goals A day in the life - Train ML models for deployment in simulation and real-world robots, identify and document their limitations post-deployment - Drive technical discussions within your team and with key stakeholders to develop innovative solutions to address identified limitations - Guide fellow scientists in solving complex technical challenges, from sim2real transfer to training RL policies - Mentor team members while maintaining significant hands-on contribution to technical solutions About the team Fauna Robotics, an Amazon company, is building capable, safe, and genuinely delightful robots for everyday life. Our goal is simple: make robots people actually want to live and interact with in everyday human spaces. We believe that future won’t arrive until building for robotics becomes far more accessible. Today, too much effort is spent reinventing the fundamentals. We’re changing that by developing tightly integrated hardware and software systems that make it faster, safer, and more intuitive to create real-world robotic products. Our work spans the full stack: mechanical design, control systems, dynamic modeling, and intelligent software. The focus is not just functionality, but experience. We’re building robots that feel responsive, expressive, and genuinely useful. At Fauna, you’ll work at the frontier of this space, helping define how robots move, manipulate, and interact with people in natural environments. It’s an opportunity to solve hard problems across hardware and software with a team focused on making robotics accessible and joyful to build. If you care about making robotics real for everyone and building systems that are as delightful as they are capable, we’re interested in hearing from you.