Social
Conversational AI

“Ambient intelligence" will accelerate advances in general AI

Alexa’s chief scientist on how customer-obsessed science is accelerating general intelligence.

By Rohit Prasad
December 21, 2021
13 min read
Share

As the world has become more connected, and computing has permeated our surroundings, a new AI paradigm is emerging: ambient intelligence. In this paradigm, our environment responds to our requests and anticipates our needs, provides information or suggests actions, and then recedes into the background.

Rohit Prasad.jpg
Rohit Prasad, Alexa head scientist and senior vice president at Amazon.

This vision of ambient intelligence is not that different from the one on Star Trek. But for most of the last decade, the focus has been reactive assistance — for example, ensuring that customer-initiated requests to Alexa meet customers’ expectations.

In the ambient-intelligence vision, an AI service such as Alexa makes sense of the state of your environment, including devices, sensors, objects, people, and activity around you, to help you in every situation where you need assistance — either reactively (customer initiated) or proactively (AI initiated).

Realizing the ultimate potential of ambient intelligence requires Alexa to bring the best of machine-intelligence capabilities together with the best of human-intelligence capabilities, which is the barometer of general intelligence today.

The most pragmatic definition of general intelligence is the ability to (1) learn multiple tasks jointly, versus modeling each task independently; (2) continually adapt to changes within a set of known tasks, without explicit human supervision; and (3) learn new tasks directly by interacting with end users.

While these general-intelligence characteristics apply to all types of AI systems, for interactive AI services such as Alexa, two more attributes are critical: (1) multisensory and multimodal intelligence — the ability to process data from multiple input sensors (e.g., microphones, cameras, ultrasound), fuse sensor data for improved understanding of customer goals, and generate output in different modalities (e.g., speech, text, image, video); and (2) interaction skills — the ability to converse in a human-like manner, which encompasses not just command of natural language but also the ability to recognize and respond to affect.

What this means for our customers is that Alexa will become

  • More competent: Alexa’s functionalities and skills will expand much faster through multitask intelligence. Additionally, Alexa will improve through self-learning, becoming less reliant on labeled data;
  • More natural and conversational: Alexa interactions will be as free flowing as human interactions through multisensory intelligence, generalizable language models, commonsense reasoning, and affect modeling; 
  • More personalized: Alexa will adapt to each individual using speech and computer vision. Further, customers will be able to directly personalize Alexa explicitly and implicitly;  
  • More insightful and proactive: Alexa will anticipate customer needs through awareness of the shared environment, make suggestions, and even act on customers’ behalf;  
  • More trustworthy:  Alexa will have the same attributes that we cherish in trustworthy people, such as discretion, fairness, and ethical behavior.

In the past year, Alexa has made considerable progress on all these fronts.

More competent

Alexa receives billions of requests per month, and it is critical for it to answer each of these requests to customers’ satisfaction. In 2021, through advances in automatic speech recognition (ASR), natural-language understanding (NLU), and action resolution, Alexa has become 13% more accurate than the previous year — even as the complexity of customer requests has increased.

Alexa has more than 130,000 third-party skills, whose diversity is a testament to their developers’ creativity. Further, it is available in more than 15 language variants across more than 80 countries, most recently Khaleeji Arabic in Saudi Arabia.

Through advances in large pretrained language models, we are making it easier to expand Alexa’s functionality in terms of both skills and languages. Specifically, we have trained an “Alexa Teacher Model,” a large, pretrained, multilingual model with billions of parameters that encodes language as well as salient patterns of interactions with Alexa. Instead of building new task-specific NLU models (e.g., a skill, a feature, or a language) from scratch on task-specific data, we can build them by fine-tuning the Alexa Teacher model, which provides substantial gains in performance from the same amount of task-specific training data.

While today, the Alexa Teacher Model itself is impractical for real-time language understanding, once it is distilled and fine-tuned, it is compact enough to run in real time but remains more accurate than a similar-sized model trained from scratch. The capacity to generalize across tasks, which the language model enables, is one of the hallmarks of general intelligence.

ATM pipeline.png
The Alexa Teacher Model (AlexaTM) pipeline. The Alexa Teacher Model is trained on a large set of GPUs (left), then distilled into smaller variants (center), whose size depends on their uses. The end user adapts a distilled model to its particular use by fine-tuning it on in-domain data (right).

Models derived from the Alexa Teacher Model have helped reduce customer friction in several locales and will help facilitate and scale multilingual and multimodal use cases in coming years.

Still, faster deployment of new functionality is not sufficient. Customer interactions with Alexa are ever evolving, so Alexa needs to improve continuously. To that end, we have expanded Alexa’s self-learning capability — in particular, its ability to automatically learn from implicit feedback, e.g., when a customer cuts Alexa off in order to rephrase a query.

Currently, we have two methods for learning from implicit feedback. One is a mechanism that learns to automatically reformulate the ASR output to ensure a more accurate response, and the other automatically annotates interaction data to enable the retraining of NLU models with minimal human involvement.

At this year’s Conference on Empirical Methods in Natural Language Processing (EMNLP), Alexa AI researchers presented papers reporting our progress on both these fronts.

Learning how to rewrite customer requests requires identifying which successful requests are rephrases of unsuccessful ones. Past work on rephrase detection considered sentences in pairs, determining the likelihood that one is a rephrase of the other. In our EMNLP paper, we explain how to use temporal features of the dialogue history to better identify rephrases, with an accuracy improvement of 28% on one test dataset.

Rephrases.png
Earlier rephrase detection models computed similarity scores between pairs of queries (right), which could lead to inaccuracies. A new model instead uses full dialogue context (left) to more accurately detect rephrases by leveraging session-level semantic information. From “Contextual rephrase detection for reducing friction in dialogue systems”.

In the other paper, we describe a scalable framework for using automatically annotated data to continually update our NLU models. This paper shows how to operationalize our previous work on automatic annotation, to deliver immediate results to our customers.

More natural and conversational

As magical as it is to interact with Alexa by simply saying its name, repeating the name during longer interactions feels unnatural: when we’re talking to other people, we don’t use their names on every turn.

This year, we took a major step toward making interactions with Alexa more natural through Conversation Mode, which leverages Echo Show 10’s camera to enable wake-word-free interactions by improving the detection of device directedness (i.e., the intent of addressing Alexa) — even when there are multiple people in the room, conversing with each other as well as with Alexa.

Conversation Mode uses novel computer vision algorithms to gauge customers’ physical orientations toward the device, which indicate whether they’re addressing Alexa or each other. The combination of visual and audio information dramatically improves device-directed-speech detection relative to either modality used independently. Further, on-device speech recognition using fully neural recurrent-neural-network transducers ensures that Alexa recognizes conversational speech with low latency.

We have also started extending Alexa’s conversational memory, going beyond anaphoric references within an interaction session (e.g., “What is its resolution?” while shopping for TVs) to temporarily maintain memory across sessions in certain situations. For example, for high-consideration purchases such as TVs, Alexa remembers your last interaction and starts off your next interaction where you left off. This capability required us to extend Alexa Conversations, which trains deep-learning-based models on synthetic data automatically generated from a small amount of developer-provided data.

As effective as large neural transformer-based language models are for generating textual responses, they lack the commonsense and knowledge grounding they need to be truly useful in large-scale human-machine interactions. This fall, to help foster the type of invention needed to overcome these challenges, we released the commonsense dialogue dataset, which consists of more than 11,000 newly collected dialogues. In each dialogue, successive turns are related by relationship triples in the public commonsense knowledge graph Conceptnet, such as <doctor, LocateAt, hospital> or <specialist, TypeOf, doctor>.

Commonsense dialogue.png
In each dialogue in the commonsense-dialogue dataset, successive turns are related by relationship triples in the public commonsense knowledge graph Conceptnet, such as <piano, RelatedTo, musical> or <musical, RelatedTo, violin>.

Another way to inject common sense into dialogue models is to enable them to import information from online or other sources as needed, on the fly. At the NeurIPS Workshop on Efficient Natural Language and Speech Processing (ENLSP) earlier this month, Alexa researchers won a best-paper award for doing just that. They propose a few-shot-learning approach to training a knowledge-seeking-turn detector, which can recognize customer questions that can’t be answered through existing API calls.

This year, we also published several papers on affect modeling. At the International Conference on Acoustics, Speech, and Signal Processing, we presented the use of contrastive unsupervised learning to improve emotion recognition when training data is scarce; and at the Spoken Language Technologies conference, we described the adaptation of pretrained language models, which have been so successful at natural-language-processing tasks, to the problem of social and emotional commonsense reasoning.

On the flip side, when human speakers recognize shifts in the emotional states of people they’re talking to, they modify the affect in their responses. At the Speech Synthesis Workshop (SSW11) this summer, we extended our previous work on prosody variation to modify the affective characteristics of synthesized speech.

More personalized

AI’s ability to conform to customers as opposed to the other way around differentiates it from other technological advancements. This fall, we launched multiple new services that allow our customers to personalize AI in a self-serve fashion.

With preference teaching, customers can explicitly teach Alexa which skills should handle weather-related questions, which sports teams they follow, and which cuisines they prefer.

CustomAED_embedding.png
A two-dimensional projection of embeddings produced through Custom Sound Event Detection. New sounds are identified by their location in the embedding space.

With Custom Sound Event Detection, customers can train Alexa to recognize new sounds — such as a doorbell ringing — from just a handful of examples. Custom Sound Event Detection uses proximity in a neural network’s representational space to recognize instances of the same sound.

Custom Event Alerts for Ring Video Doorbell cameras and Spotlight cameras works in a similar way. With just a few examples, customers can train their devices to recognize certain states of affairs in the world — such as a shed door that has been left open.

In August, we introduced adaptive volume for Alexa, which lets Echo devices adjust their volume according to ambient-noise levels, so that the perceived noise level stays consistent for the customer. One of the key elements of the approach is algorithmically separating the speech signal and the noise signal, so that they’re separate inputs to the volume adaptation model.

We also launched adaptive listening for US English, an opt-in feature that gives customers more time to finish speaking before Alexa responds, making Alexa a more accessible, patient listener. For speakers with certain speech impediments, adaptive listening has reduced the friction in their Alexa interactions by more than two-thirds.

Finally, Alexa customers can choose to interact with celebrity personalities such as Amitabh Bachchan, Melissa McCarthy, Samuel L. Jackson, or Shaquille O'Neal. At the end of the year, we even brought holiday cheer to Alexa interactions by launching the festive personality of Santa Claus.

More insightful and proactive

Today, one in four smart-home interactions is initiated by Alexa, due to the expansion of its predictive and proactive features such as hunches and routines.

Since 2018, Alexa hunches have recognized anomalies in customers’ daily routines and suggested corrections — noticing that a light was left on at night and offering to turn it off, for instance. This year, we gave customers the option of making hunches more proactive, so Alexa can act on their behalf. When proactive hunches are enabled, Alexa will turn that light off for you without asking first.

Routines let you initiate a sequence of actions with a single trigger word, rather than issuing the same instructions over and over again. Previously, customers had to specify which actions they wanted to string together. But this year, we began phasing in inferred routines. With inferred routines, Alexa recognizes sequences of actions that customers commonly repeat — such as, say, turning on the kitchen lights, starting the coffee maker, and playing the “Wake Up!” playlist — and suggests combining them into a routine. To save the routine, the customer simply accepts Alexa’s suggestion.

We have also continued to expand latent-goal prediction, where Alexa recognizes the larger customer need implied by an initial request and suggests actions or skills to fulfill that need. For instance, a customer asks, “Who won the Celtics game?”, and after answering, Alexa asks, “Would you like to know when the Celtics are playing next?”

Latent-goal prediction uses pointwise mutual information to measure the likelihood of an interaction pattern in a given context relative to its likelihood across all Alexa traffic, and it uses bandit learning to track whether recommendations are helping or not and suppress underperforming experiences.

We have also introduced visual ID on our latest Echo device, Echo Show 15. With visual ID, Alexa shows notes and other reminders just for you (e.g., “Leave a note for Jack that his new passport has arrived”). Visual ID is also available on Astro, an Alexa-enabled home robot that extends environment and state awareness to your physical space. Astro can follow you playing media or find you to deliver calls, messages, timers, alarms, or reminders. With a Ring Protect prosubscription, Astro can also proactively patrol your home and investigate anomalous activities.

More trustworthy

Preserving customer privacy is an uncompromisable tenet for us and an invention area. Differential privacy in particular is one of our key areas of focus. This year, we won a best-paper award at the annual meeting of the Florida Artificial Intelligence Research Society (FLAIRS) for an approach to improving the performance of machine learning models while still meeting the privacy standards imposed by differential-privacy analysis.

At the Conference of the European Chapter of the Association for Computational Linguistics, we presented a method for protecting privacy by automatically rephrasing training text while preserving their semantic sense, in a way that, again, meets differential-privacy standards.

Biased language models still.jpg
Alexa AI researchers constructed a dataset of more than 23,000 text generation prompts, each consisting of six to nine words of a sentence on Wikipedia. The prompts can be used to test language models for bias.
Credit: Glynis Condon

We want Alexa to work equally well for everyone. To that end, in addition to our partnership with the National Science Foundation in the area of fairness in AI, we are pursuing research into detecting and mitigating inappropriate bias. At the ACM Conference on Fairness, Accountability, and Transparency (FAccT) and the Conference of the European Association for Computational Linguistics, we published a pair of papers on measuring bias in language models and detecting bias in datasets for training models that recognize unreliable news.

The path ahead

I recognize that there are multiple paths to general AI, each with years of fundamental research ahead of it. I believe Alexa and its underlying vision of ambient intelligence offer a pragmatic path to general AI— one where every advancement makes Alexa more useful for our customers in their daily lives.

I am in awe at the rate of invention from the Alexa team in the most difficult circumstances. As we wrap up yet another year of the COVID pandemic, I hope the advances the worldwide community of AI researchers is making in every discipline of AI will help us prevent future pandemics.

Research areas
Tags
About the Author
Rohit Prasad
Rohit Prasad is a senior vice president and head scientist of Amazon's Artificial General Intelligence organization.

Related content

Work with us

See more jobs See more jobs
Research Scientist, Pricing Science
US, WA, Seattle
Pricing is one of the most consequential decisions Amazon makes, and the science behind it needs to be causally rigorous, not just predictive. The P2 Optimization Science team builds the machine learning (ML) systems that power Amazon's pricing decisions at scale: demand lift models, customer lifetime value (LTV) frameworks, and the experimentation infrastructure that validates whether pricing changes actually work. We're looking for a Research Scientist to own causal inference at the intersection of ML and pricing experimentation. Our team has identified a real gap: the methodological bridge between econometric analysis and production-scale ML pipelines needs a practitioner who lives in both worlds. You'll build conditional average treatment effect (CATE) estimation models, design analysis workflows for pricing experiments, and develop reusable causal ML infrastructure that the broader team — including non-ML scientists — can rely on. If you're a scientist who wants to tackle hard causal identification problems in a high-stakes production environment, and who finds satisfaction in making rigorous methods accessible to a broader team, this role is for you. Key job responsibilities - Design, train, evaluate, and deploy end-to-end causal ML pipelines for pricing use cases, including CATE estimation models and heterogeneous treatment effect frameworks. - Serve as the team's causal ML methodology expert — defining identification strategies, model selection criteria, evaluation standards, and the tradeoffs between econometric and ML approaches to causal estimation. - Contribute causal analysis methodology to pricing A/B test post-analysis and build reusable tooling that economists and other non-ML scientists can adopt without requiring deep ML expertise. - Assess emerging causal inference methods (e.g., synthetic difference-in-differences, generalized random forests, causal representation learning) for applicability to Amazon's pricing context, and author internal methodology proposals for adoption. - Partner with senior economists on identification strategy and causal assumptions, collaborate with software and data engineering partners on production deployment, and align with product managers on experiment design requirements. A day in the life Your work directly shapes the prices customers see on hundreds of millions of Amazon products. In a given week you might investigate an optimization anomaly in simulation and trace it to an unmodeled market dynamic, then design an offline evaluation framework to benchmark competing approaches before committing to online testing. You'll collaborate with senior economists on identification strategy for a pricing experiment, present a science proposal for incorporating a new competitiveness signal into an optimization system, or work with the experimentation platform team on randomization design. You'll also have opportunities to write up novel scientific findings for submission to venues such as KDD, NeurIPS, or the ACM Conference on Economics and Computation. About the team The P2 Optimization Science team sits at the center of Amazon's pricing decisions. We build the ML systems, causal inference pipelines, and experimentation infrastructure that determine how prices are set across Amazon's global catalog. Our team brings together research scientists, economists, and engineers who collaborate closely to turn rigorous methodology into production-quality systems with measurable business impact. We value scientific rigor, clear communication across disciplines, and a culture where every team member's work is designed to be extensible and useful beyond their own project. If you want your science to directly influence how hundreds of millions of customers experience Amazon, this is the team to join.
Applied Scientist, Center for Quantum Computing
US, CA, San Francisco
Amazon’s Center for Quantum Computing (CQC) is building the next generation of quantum computing hardware. The Center for Quantum Computing (CQC) is our longest-horizon bet and operates at the intersection of hardware and physics. The goal is a quantum computer that could solve in hours what today's most robust machines would take thousands of years to compute, unlocking breakthroughs in myriad of areas ranging from advanced materials for batteries to cryptography and fundamentally-secure data sharing protocols. This program involves R&D in device theory, materials science, nano-scale fabrication, superconducting circuit design, cryogenic refrigeration, and quantum algorithms and applications. Currently, the quantum hardware team is focused on improving the underlying chip fabrication of the quantum processor and the scaling of the control system needed to run large-scale quantum circuits. As part of this effort, the team is expanding its chip fabrication capabilities through the construction of a new cleanroom facility dedicated to superconducting quantum chip fabrication, and working with strategic partners to develop scaled versions of the control electronics, I/O wiring, and cryogenic refrigeratation needed for utility-scale quantum computers. We operate with a startup-like pace inside one of the world's most capable technology organizations, combining the urgency of a competitive frontier with the resources and infrastructure of pan Amazon. We are seeking an Applied Scientist III to join our team and drive the design, construction, and operation of quantum processors. You will work alongside world-class physicists and engineers to push the frontier of quantum computing — from laser system design and atom trapping to high-fidelity gate execution and quantum error correction. This is a rare opportunity to be part of shaping a technology that could define the next era of computing. Key job responsibilities - Design, build, and operate quantum computing experiments, including optical trapping arrays, laser cooling and state-preparation systems, and high-fidelity gate-driving optics - Develop and optimize two-qubit entangling gates using interactions, targeting fidelities at or beyond the current state of the art (>99.5%) - Implement and characterize erasure-detection protocols leveraging the metastable-state structure of ¹⁷¹Yb to achieve hardware-native erasure conversion rates >90% - Contribute to quantum error correction experiments, including surface codes and quantum low-density parity-check (qLDPC) codes, and work toward demonstrating error suppression factors (Λ) well beyond current published records - Design and improve atom-array loading, rearrangement, and continuous replenishment techniques to support registers of thousands to tens of thousands of atoms - Develop control software and calibration routines for high-speed experimental sequences, including real-time feedback and adaptive protocols - Analyze experimental data, build physical models to explain observed behavior, and propose and execute improvements to system performance - Collaborate with theory, software, and hardware engineering teams to co-design quantum architectures, error-correcting codes, and decoder systems - Publish results in leading scientific journals and present at top conferences to maintain Amazon's visibility in the quantum research community
Imitation Learning Engineer, RIVR
CH, Zurich
RIVR, an Amazon company, is building Physical AI by deploying autonomous robots for real-world doorstep delivery. Operating daily in diverse urban environments, RIVR's robots continuously learn from and navigate the millions of scenarios encountered during deliveries. By owning the full stack from software to hardware, RIVR is purpose-built for safety, reliability, and the customer from day one. Imitation learning enables our robot to mimic "expert" behaviors, derived from human demonstrations or algorithmic strategies. By utilizing state-of-the-art generative AI and similar methods, our wheeled-legged robot can significantly enhance its autonomy and manipulation skills. In this role, you will enable robots to autonomously generate actions from demonstrations and real-time sensor data. These processes may also incorporate responses to natural language commands, further advancing the robot's skills. We are seeking an expert in imitation learning and generative AI techniques that directly produce robot behaviors, along with a deep knowledge of both supervised and self-supervised learning algorithms. If you are passionate about pushing the boundaries of AI and eager to deliver innovative solutions, we invite you to join us in shaping the future of intelligent robotics. Key job responsibilities Develop imitation learning algorithms, such as diffusion policies, to enable robots to autonomously execute actions based on demonstrations and real-time sensor data. Design, test, and refine your algorithms to meet the demands of complex real-world autonomy and manipulation tasks. Construct a dataset for the imitation learning algorithm using human demonstrations or automated expert algorithms. Streamline the workflow to efficiently expand the imitation learning dataset with new tasks. Collaborate with the reinforcement learning team to innovate methods that leverage both simulated and real-world data. Implement deployment-ready code for the real robot, optimized for the robot’s computational constraints. Build, lead and mentor an exceptional team of software engineers. Provide expert guidance to product managers and executives for strategic decision-making. Create and maintain documentation, guidelines, and best practices to streamline knowledge sharing.
Principal Applied Scientist, Ads Measurement Science
US, NY, New York
The Measurement Science team in the Measurement, Ad Tech, and Data Science (MADS) organization of Amazon Ads serves a centralized role developing solutions for a multitude of performance measurement products. We create solutions which measure the comprehensive impact of their ad spend, including sales impacts both online and offline and across timescales, and provide actionable insights that enable our advertisers to optimize their media portfolios. We leverage a host of scientific technologies to accomplish this mission, including Generative AI, classical ML, Causal Inference, Natural Language Processing, and Computer Vision. This Principal Applied Scientist leads the technical vision and execution for Amazon Advertising's performance measurement. Our measurement products and tools help customers understand the holistic impact of their spend on Amazon Ads and optimize their portfolios. This is a complex, multi-disciplinary scientific space spanning machine learning, causal inference, statistics and experimentation, natural language processing, and generative AI. As a GenAI-first organization, we build foundational and agentic models that power advertiser use cases across Ads, while empowering our Applied Scientists to directly build and ship products. You will guide a team of scientists with expertise across different domains to come together and innovate on some of the hardest problems in advertising, made more challenging by evolving privacy regulations and technological changes. In measuring the comprehensive impact of advertising, key challenges you will tackle include capturing impact: 1) both for identified and increasingly unidentified web traffic in light of an evolving global privacy landscape; 2) for outcomes across retailers, including both within and beyond Amazon; and 3) for both short- and long-term impacts of advertising. Key job responsibilities * Driving the scientific vision of the teams in your organization and you advise and influence its technical leadership on performance measurement models and products. * Identifying, tackling, and proposing innovative solutions to intrinsically hard, previously unsolved problems. You solve problems pragmatically, applying judgment and experience to balance trade-offs between competing interests. * Bringing clarity to complex problems, probing assumptions, illuminating pitfalls, fostering shared understanding, and guiding towards effective solutions. * Serving and being recognized by internal and external peers as a thought leader in modeled measurement in Ads. You will engage routinely with the broader scientific community, e.g. as a reviewer, program committee member, workshop organizer, or chair at peer-reviewed internal/external conferences. * Leading by example, with your papers, analyses, and code setting the standard in your organization for scientific engineering excellence. * Leading science and design reviews, aligning teams across your organization around coherent algorithmic strategies. * Influencing your team’s science and business strategy by driving one or more team roadmaps contributing to the organization’s roadmap and taking responsibility for some organizational goals. You drive multiple new product features from inception to production launch. * Guiding the career development of others, actively mentoring and educating the larger applied science community on trends, technologies, and best practices. You drive the documentation of scientific innovation. You attract bar-raising talent from academia and industry. * Staying abreast of academic and industry trends, knowing the state of the art, and building on top of existing solutions instead of reinventing when possible. About the team We are a team of scientists across Applied, Research, Data Science and Economist disciplines. You will work with colleagues with deep expertise in ML, NLP, CV, Gen AI, and Causal Inference with a diverse range of backgrounds. We partner closely with top-notch engineers, product managers, sales leaders, and other scientists with expertise in the ads industry and on building scalable modeling and software solutions.
Robotics Applied Scientist
US, CA, Sunnyvale
Amazon is seeking exceptional talent to help develop the next generation of advanced robotics systems that will transform automation at Amazon's scale. We're building robotic systems that combine frontier AI, sophisticated control, and advanced mechanical design to create adaptable automation capable of working safely alongside humans in dynamic environments. This is an opportunity to shape the future of robotics and automation at an unprecedented scale, working with world-class teams pushing the boundaries of what's possible in robotic manipulation, locomotion, and human-robot interaction. We are seeking an Applied Scientist to develop tactile- and force-driven manipulation policies for highly dexterous, multi-fingered robotic hands and grippers. In this role you will research and build policy learning algorithms that enable robotic hands to perform grasping and in-hand manipulation in unstructured, real-world environments, using touch and force feedback as first-class signals rather than afterthoughts. Much of this work is grounded in reinforcement learning. To be successful you need to be highly motivated, dive deep, and deliver to the highest standards. You will demonstrate strong working knowledge of modern policy learning methods, with expertise in taking algorithms from simulation onto real hardware. That means being comfortable working against real physical constraints and treating sim-to-real as a core research problem rather than optimizing for benchmark numbers alone. You will work across tactile signal processing, contact-rich manipulation, sim-to-real transfer, and multi-modal sensor fusion to build robust solutions for autonomous grasping, dexterous re-orientation, and fine motor control. You will bring the desire to learn from new challenges, and the problem-solving and communication skills to work within a highly interactive and experienced team. Key job responsibilities - Design and train manipulation policies that leverage tactile and force feedback for dexterous grasping, in-hand manipulation, and contact-rich tasks, using reinforcement learning, imitation learning, or hybrid approaches. - Drive sim-to-real transfer, closing the gap between simulated and physical performance on real robotic hands. - Shape objectives and control strategies with the hardware in mind, accounting for how motors and actuators actually behave: transmission ratios, torque versus power, backdrivability, and thermal limits. - Build simulation-based and on-robot evaluation frameworks, with benchmarks and metrics that make tactile perception and policy performance systematically comparable across iterations. - Own scientific and technical projects within the tactile sensing and manipulation workstreams, driving from research concept through deployment on physical robotic systems — and, at the senior level, set direction across a workstream. - Collaborate with hardware, mechanical design, firmware, and controls teams — and with partner applied science organizations — to translate research advances into deployable robotic capabilities, and to inform sensor and actuator design decisions with policy-level requirements. - Publish at top-tier venues and build collaborations with the external research community. - Contribute to a strong scientific bar on the team through code and design review, and mentor engineers and interns working on real-world manipulation and sensing problems. A day in the life Your morning might start with a standup alongside hardware and controls engineers reviewing overnight sim-to-real training runs on a multi-finger gripper, debugging why a grasp policy that worked in simulation is slipping on a real sensor array. Mid-morning you join the weekly tactile sensing workstream to align on sensor integration milestones and share early results from a contact-state estimator you have been prototyping. After lunch you spend a focused block iterating on a reinforcement learning reward formulation, testing variations in simulation before queuing runs on the cluster. Later you pair with a mechanical engineer to review sensor placement trade-offs on an upcoming gripper revision, then wrap the day by drafting a short experiment write-up and sketching next steps for a conference submission. No two days look exactly the same: one week you may be collecting teleoperated demonstrations on the physical robot to seed an imitation learning pipeline; the next you could be deep in a codebase refactor to support a new tactile modality. Throughout, you balance hands-on research — writing algorithms, running experiments, analyzing data — with cross-team collaboration and mentoring. The common thread is moving from research insight to working capability on real hardware, with a tight feedback loop between simulation and the physical world. About the team Our is developing next-generation manipulation capabilities for highly dexterous multi-fingered grippers and hands. Our work spans tactile sensor development, manipulation policy learning, and the tight integration of sensing hardware with intelligent control software. We are building systems that can feel, adapt, and manipulate with human-level dexterity. You will join a team working at the frontier of tactile sensing and contact-rich manipulation, helping define how robots perceive and interact with objects through touch. Because we build the hands as well as the policies that drive them, you will have a direct line to the sensor, actuator, and mechanism designs your algorithms depend on — and real influence over how they evolve. This is an opportunity to solve hard problems across perception, learning, and physical interaction alongside a team of scientists and engineers building toward real-world deployment.
Senior Applied Scientist, Amazon Leo, Amazon Leo Networking Software
US, WA, Redmond
Amazon Leo is building a low Earth orbit satellite broadband network to deliver fast, affordable connectivity to customers around the world, including those underserved by traditional infrastructure. Operating a live constellation and its ground network at scale means service disruptions are an inevitable reality, and how quickly we detect them, understand them, and prevent them from recurring defines the trust we earn as a connectivity provider. We are seeking a talented Senior Applied Scientist to build the next generation of observability and availability science for the Leo network. Leo already detects outages, notifies affected customers, and restores service. The frontier is prediction and prevention: reasoning over complex network and ground-segment telemetry to autonomously detect anomalies before they reach customers, attribute root cause automatically across a sprawling event surface, and rationalize a large, still-maturing metric set so responders trust what the system tells them. In this role you will work backwards from the needs of customers and operations teams. You will own the scientific direction end to end, partnering closely with network engineers, the Network Operations Center, and the Incident Management team, and the Software and Networking orginization to translate a long-term science vision into concrete research and delivery roadmaps. Export Control Requirement: Due to applicable export control laws and regulations, candidates must be a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum. Key job responsibilities * Own the observability and availability science end to end, from long-term vision to research roadmap to production delivery, working backwards from the needs of customers, the NOC, and network engineering. * Develop time-series and graph-based models with dynamic baselining that detect network and ground-segment anomalies before they cross the High Severity threshold, enabling proactive investigation rather than reactive response. * Reason over complex, multi-system telemetry to attribute cause automatically during live HSEs and LSEs, so Call Leaders and engineers spend time mitigating, not hunting. * Provide the science behind the network's bucketed availability decomposition (geometric and satellite coverage, ground and network segments, customer terminals, payload, and planning), attributing every movement to a true cause rather than noise, so leadership steers on signal and engineering effort lands where it actually moves availability and throughput. * Deliver the models, analyses, and metrics that feed the daily system readiness and validation review consumed by senior leadership and product and engineering, turning availability and throughput data into a defensible, evidence-based read on progress against GA entry criteria. * Own the recurring, fleet-wide "hot topic" investigations that span ASIC, firmware, payload, network, and planning, separating correlated symptoms from true root cause across thousands of nodes, quantifying how much availability or throughput each is costing, and answering whether it is a launch-blocking risk and whether it is improving. * Deliver alert correlation and deduplication, false-positive reduction, and metric-quality scoring so responders trust what the system tells them, directly supporting a large, still-maturing KPI set. * Tie your science to the unit the org actually measures: reduce detection and attribution time, raise availability against the network's bar, and increase the share of events prevented before customer impact. * Partner day to day with the Network Operations Center, Network Engineering, and the Incident Management (Call Leader) program; sit on live HSE/LSE bridges as the science partner and turn event learnings into model improvements. * Feed COEs and RCAs back into detection and attribution so recurring signatures are caught earlier, and help design and score gamedays that stress-test both the network and your models against real, closed events. * Build and deploy production pipelines on AWS (e.g., SageMaker, Bedrock, Glue) with model validation, monitoring, drift detection, and explainability, because operations teams must be able to trust and act on model output under pressure. * Set data-science standards, mentor scientists, and drive best practices across the Leo networking organization as the science function scales. * Produce explainable output and written narratives that let responders and leaders three or more levels up act on your models during and between events.
Data Scientist, OTS - Data ANCHOR Team
US, TX, Austin
The OTS ANCHOR (Analytics, Insights, and Centralized Hub for OTS Reporting) team is part of the WW IT Operations & RME organization, responsible for delivering data solutions that drive operational excellence across Amazon's global IT infrastructure. We build data pipelines, predictive models, develop analytical intelligence frameworks, and create AI-powered solutions that transform how Amazon manages, triages, and resolves IT operational incidents at scale pointing towards the Letting Data Tell the Story north start vision. We are seeking a Data Scientist to help own the end-to-end data science charter for critical operational workstreams — from forecasting IT service demand and capacity planning to designing intelligent severity triaging models and building the analytical backbone for agentic AI solutions. You will work backwards from complex operational problems to create models and solutions that directly impact the efficiency and reliability of Amazon's IT operations worldwide. Key job responsibilities - Design, develop, and deploy machine learning models for IT operations forecasting, including incident volume prediction, capacity planning, and workforce demand modeling - Build and maintain severity classification and intelligent triaging models that process thousands of daily IT service tickets across global regions - Develop anomaly detection systems to identify emerging patterns in operational data, supporting proactive incident prevention - Apply econometric and statistical methods to measure the impact of operational interventions and inform resource allocation decisions - Partner with engineering teams to productionize ML models via automated pipelines using AWS services (SageMaker, Step Functions, Lambda, Redshift) - Design and conduct experiments to empirically validate model performance and operational impact - Contribute to data quality monitoring frameworks and help maintain standards for operational data integrity - Collaborate with cross-functional teams (ServiceNow platform, Decision Intelligence, Field Operations) to translate analytical insights into actionable recommendations - Communicate findings, model performance, and recommendations to stakeholders and senior leaders through written documents and presentations - Contribute to the team's science roadmap, identifying opportunities to apply ML/AI (including Generative AI) to operational challenges A day in the life Amazon Benefits: Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: 1. Medical, Dental, and Vision Coverage 2. Maternity and Parental Leave Options 3. Paid Time Off (PTO) 4. 401(k) Plan
2027 Applied Science Internship - Robotics - United States
US, MA, North Reading
Robotics Applied Scientist II Intern/Co-op (2027) Are you passionate about robotics and research? Do you want to solve real customer problems through innovative technology? Do you enjoy working on scalable research and projects in a collaborative team environment? Do you want to see your science solutions directly impact millions of customers worldwide? We’re looking for current PhD students with a passion for robotic research and applications to join us in 2027 as Robotics Applied Scientist II Interns/Co-ops to shape the future of robotics and transform automation at Amazon’s scale. By applying to this role, you will be considered for Robotics Applied Scientist II Intern/Co-op (2027) opportunities across various Robotics teams at Amazon with different robotics research focus. Internship positions are available for multiple locations, durations (3 to 6+ months), and year-round start dates (winter, spring, summer, fall). Join us at the forefront of Applied Science, where your contributions will shape the future of Robotics! Key job responsibilities As a Robotics Applied Scientist II Intern/Co-op, you will work alongside global experts to advance the state of the art in robotics and artificial intelligence, tackling innovative problems that bridge the gap between research and real-world robotics implementation to innovate and build solutions on behalf of our customers. We are particularly interested in students with specialization and expertise in one or more of the following research areas: - Robotics - Artificial Intelligence - Robotics Manipulation (e.g., robot arm, grasping, dexterous manipulation, end of arm tools/end effector) - Autonomous Mobile Robots - Mobile Manipulation - Multi-Robot Coordination - Autonomous Navigation - Locomotion - Motion/Path Planning - Controls - Perception - Sensing - Robot Learning - Machine Learning - Computer Vision - Large Language Models (LLM), Vision-Language-Action Models (VLA), Vision-Language-Models (VLM) - World Models - Human-Robot Interaction - Robotics Simulation - Optimization - And related! Key responsibilities may include: - Develop novel, scalable algorithms and/or modeling techniques that advance the state-of-the-art in science areas such as robotics, artificial intelligence, or related - Tackle challenging, innovative research problems within robotics, artificial intelligence, or related that bridge the gap between theoretical advancement and practical robotics implementation - Collaborate with cross-functional teams to solve complex problems, and to communicate research findings clearly, leveraging your expertise in robotics, artificial intelligence, or related - Demonstrate the ability to work independently, thrive in a fast-paced, ever-changing environment - Leverage AI-powered tools where applicable to accelerate research, experimentation, and prototyping. - Critically review and validate outputs from AI tools and automated systems. About the team Explore Robotics Teams at Amazon At Amazon, robotics is where innovative research meets real-world deployment. There are many different specialized robotics teams, including the few highlighted below. Learn more about Robotics at Amazon here: https://www.amazon.science/research-areas/robotics Fulfillment Technologies and Robotics We build dynamic partnerships between people and intelligent machines. This intricate collaboration helps Amazon fulfill orders with unmatched accuracy. Working in symphony with our robotic technology, you’ll have the opportunity to extend your technical capabilities by working alongside some of the industry’s most advanced technologies. This includes our fleet of autonomous mobile robots, sophisticated control software, and technologies like language perception, machine learning, object recognition, and semantic understanding of commands. These technologies help us deliver an ever-improving customer and employee experience, as well as improve the safety of our facilities. Amazon Robotics Amazon Robotics is at the epicenter of robotics innovation. We design, test, and build mobile robots and mechatronic systems. Our offices, labs, test floors, and factories are close together, which means we can design the hardware, write the code, test, build, and watch it all come to life. Our areas of focus include advancements in autonomous mobility, AI, robotic management software, and much more. Fauna Fauna is focused on building capable, safe, and genuinely delightful robots for everyday life. Fauna’s goal is simple: make robots people actually want to live and interact with everyday. Fauna develops tightly integrated hardware and software systems that make it faster, safer, and more intuitive to create real-world robotic products. Fauna’s work spans the full stack: mechanical design, control systems, dynamic modeling, and intelligent software. The focus is not just functionality, but overall experience. Amazon Fulfillment Technologies Amazon Fulfillment Technologies develops tech solutions that create network optimizations for all Amazon sites worldwide. We own network infrastructure and devices in all buildings and labor planning and scheduling technology for fulfillment centers in North America and Europe.
Principal Applied Scientist, FAIM (Full Funnel Agentic Intelligence and Models)
US, NY, New York
The FAIM (Full Funnel Agentic Intelligence and Models) builds the agentic AI systems behind next-generation advertising. The team develops multimodal foundation models and LLM-driven agents that power shopper understanding, audience-aware creative generation, and agentic advertising workflows. As a Principal Applied Scientist for the FAIM team, you will invent the models that help advertisers achieve desired outcomes. Today's state of the art models can setup ad campaigns to a specification, but do not understand advertising strategy well enough to help with abstract goals that real advertisers ask about like "how can I make more money." This is a rare charter to build foundational optimization science where little exists today, spanning campaign recommendation, cross-product budget allocation, incrementality measurement, and long-term-sales modeling. You will set technical direction for a growing team, partner with engineering and product to take models from research to production at Amazon scale, and directly move advertiser ROAS and new-to-brand growth. In this role you will - Own the science vision for developing evaluations of the Ads Agent - Partner with scientists from across Ads to develop best practices for evaluating agent performance both online and offline - Raise the science bar for all agent evaluations developed in Ads. - Own establishing a data driven foundation for making decisions about when new features are ready to launch in Ads Agent. Key job responsibilities - Define the long-term scientific vision for agent evaluations in Ads - Translate ambiguous advertiser needs and competing objectives into a concrete science roadmap. - Invent, prototype, and productionize machine learning and optimization solutions for joint budget allocation across sponsored ad products, spanning the shopper journey from awareness to purchase. - Develop rigorous approaches to incrementality measurement and long-term-sales modeling that ground optimization in true advertiser value. - Design and lead large-scale experiments and analyses to validate hypotheses and guide product direction. - Partner closely with engineering and product to define technical contracts, data schemas, and serving systems that carry models into production. - Raise the technical bar across science and engineering through mentorship, design reviews, and hands-on collaboration. - Grow scientific talent and publish impactful research internally and at top-tier venues.
Data Scientist, WWGS SSG Strategic Initiative Development
CN, 31, Shanghai
As WWGS is committed to building an AI-native, agentic organization, the heart of SSG's evolution is to reinvent how we engage sellers with an AI-first mindset. Delivering that ambition at scale demands breakthrough innovation in our foundational capabilities, at ever-faster speed. To meet these priorities with the pace and focus they require, we are standing up a dedicated team, organized as a POD model - a lean, cross-functional team led by a single owner who holds end-to-end decisions and the business outcome, so we can move at the speed the business demands. Our priority is to build the capability to reach the right sellers at scale, anchored on two pillars: Seller Tagging and Good Contact. Through Seller Tagging, we will turn today's fragmented, siloed seller signals into a unified seller intelligence capability that deepens and broadens how well we truly understand every seller. Through Good Contact, we will resolve the seller identity gaps across our channels and systems, so that every engagement and every system that depends on it, can consistently reach the right seller contact. The Data Scientist will be part of the POD team and support the team to establish structured data governance and model training framework, and trail predict models with real business use cases. Key job responsibilities Design and deliver seller segmentation and propensity models at scale — incorporating GMS, category, growth trajectory, engagement signals, and lifecycle stage. Build contact quality scoring and lifecycle management systems (coverage optimization, dormancy detection, reactivation modeling). Define success metrics, experimentation frameworks (A/B, causal inference), and measurement methodology for seller engagement interventions. Productionize ML models and data products — partner with engineering to deploy seller scores, contact quality indices, and recommendation signals. Explore LLM/GenAI applications: automated insight generation from seller data, contact intent classification, and intelligent report synthesis. Serve as the science representative in bi-weekly NBS theme reviews; present findings and proposals to theme Bar Raisers and leadership. Collaborate with BIE team members to democratize analytical outputs via dashboards and self-serve tools. Contribute to cross-marketplace seller behavior analysis supporting Global Expansion strategy (IN, KR, VN, LATAM). Evaluate, integrate, and iterate on AI systems — assess new AI/ML tools, frameworks, and third-party models for applicability to seller intelligence use cases.
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social