Social
Conversational AI

A quick guide to Amazon’s papers at ACL 2024

Work on large language models predominates, with a particular focus on model evaluation.

By Staff writer
August 12, 2024
5 min read
Share

Like the field of conversational AI in general, Amazon’s papers at this year’s meeting of the Association for Computational Linguistics (ACL) are dominated by work on large language models (LLMs). The properties that make LLMs’ outputs so extraordinary — such as their linguistic fluency and semantic coherence — are also notoriously difficult to quantify; as such, model evaluation has emerged as a particular area of focus. But Amazon’s papers explore a wide range of LLM-related topics, from applications such as code synthesis and automatic speech recognition to problems of LLM training and deployment, such as continual pretraining and hallucination mitigation. Papers accepted to the recently inaugurated Proceedings of the ACL are marked with asterisks.

Code synthesis

Fine-tuning language models for joint rewriting and completion of code with potential bugs
Dingmin Wang, Jinman Zhao, Hengzhi Pei, Samson Tan, Sheng Zha

Bug injection.png
Obtaining buggy partial code via bug injection. From “Fine-tuning language models for joint rewriting and completion of code with potential bugs”.

Continual pretraining

Efficient continual pre-training for building domain specific large language models*
Yong Xie, Karan Aggarwal, Aitzaz Ahmad

Data quality

A shocking amount of the web is machine translated: Insights from multi-way parallelism*
Brian Thompson, Mehak Dhaliwal, Peter Frisch, Tobias Domhan, Marcello Federico

Document summarization

The power of summary-source alignments
Ori Ernst, Ori Shapira, Aviv Slobodkin, Sharon Adar, Mohit Bansal, Jacob Goldberger, Ran Levy, Ido Dagan

Hallucination mitigation

Learning to generate answers with citations via factual consistency models
Rami Aly, Zhiqiang Tang, Samson Tan, George Karypis

Intent classification

Can your model tell a negation from an implicature? Unravelling challenges with intent encoders
Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov, Hwanjun Song, Hang Su, Saab Mansour

Irony recognition

MultiPICo: Multilingual perspectivist irony corpus
Silvia Casola, Simona Frenda, Soda Marem Lo, Erhan Sezerer, Antonio Uva, Valerio Basile, Cristina Bosco, Alessandro Pedrani, Chiara Rubagotti, Viviana Patti, Davide Bernardi

Knowledge grounding

Graph chain-of-thought: Augmenting large language models by reasoning on graphs
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, Jiawei Han

MATTER: Memory-augmented transformer using heterogeneous knowledge sources*
Dongkyu Lee, Chandana Satya Prakash, Jack G. M. FitzGerald, Jens Lehmann

Tree-of-traversals: A zero-shot reasoning algorithm for augmenting black-box language models with knowledge graphs
Elan Markowitz, Anil Ramakrishna, Jwala Dhamala, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan

Tree of traversals.png
An example of how the tree-of-traversals method uses a knowledge graph interface for the query “What actor played in both Inception and Interstellar?” From "Tree-of-traversals: A zero-shot reasoning algorithm for augmenting black-box language models with knowledge graphs".

LLM decoding

BASS: Batched attention-optimized speculative sampling*
Haifeng Qian, Sujan Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Anoop Deoras

Machine translation

Impacts of misspelled queries on translation and product search
Greg Hanneman, Natawut Monaikul, Taichi Nakatani

The fine-tuning paradox: Boosting translation quality without sacrificing LLM abilities
David Stap, Eva Hasler, Bill Byrne, Christof Monz, Ke Tran

Model editing

Propagation and pitfalls: Reasoning-based assessment of knowledge editing through counterfactual tasks
Wenyue Hua, Jiang Guo, Marvin Dong, Henghui Zhu, Patrick Ng, Zhiguo Wang

ReCoE construction.png
Demonstration of the process used to construct data for the reasoning-based counterfactual-editing (ReCoE) dataset. Straight lines represent data sourced from existing datasets; dashed lines denote data derived from LLM generation; zigzag lines denote data obtained through the corruption of other data. From "Propagation and pitfalls: Reasoning-based assessment of knowledge editing through counterfactual tasks".

Model evaluation

Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models
Francesco Tonolini, Jordan Massiah, Nikolaos Aletras, Gabriella Kazai

ConSiDERS—the-human evaluation framework: Rethinking human evaluation for generative large language models
Aparna Elangovan, Ling Liu, Lei Xu, Sravan Bodapati, Dan Roth

Factual confidence of LLMs: On reliability and robustness of current estimators
Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, Lluís Marquez

Fine-tuned machine translation metrics struggle in unseen domains
Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, Brian Thompson

Measuring question answering difficulty for retrieval-augmented generation
Matteo Gabburo, Nicolaas Jedema, Siddhant Garg, Leonardo Ribeiro, Alessandro Moschitti

Model robustness

Extreme miscalibration and the illusion of adversarial robustness
Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, George Karypis

Multimodal models

CaMML: Context-aware multimodal learner for large models
Yixin Chen, Shuai Zhang, Boran Han, Tong He, Bo Li

CAMML.png
The CaMML framework, which consists of a retriever, a perceiver and a generator. After receiving user query q, the CaMML retriever identifies relevant multimodal contexts C from the data store. Then the CaMML perceiver seamlessly integrates data of various modalities, effectively encoding long-context information and injecting it into the CaMML generator. This enables the prediction of responses that are conditioned on both the context and the query. From "CaMML: Context-aware multimodal learner for large models".

Multi-modal retrieval for large language model based speech recognition
Jari Kolehmainen, Aditya Gourav, Prashanth Gurunath Shivakumar, Yi Gu, Ankur Gandhe, Ariya Rastrow, Grant Strimel, Ivan Bulyko

REFINESUMM: Self-refining MLLM for generating a multimodal summarization dataset
Vaidehi Patil, Leonardo Ribeiro, Mengwen Liu, Mohit Bansal, Markus Dreyer

Ordinal classification

Exploring ordinality in text classification: A comparative study of explicit and implicit techniques
Siva Rajesh Kasa, Aniket Goel, Sumegh Roychowdhury, Karan Gupta, Anish Bhanushali, Nikhil Pattisapu, Prasanna Srinivasa Murthy

Question answering

Beyond boundaries: A human-like approach for question answering over structured and unstructured information sources*
Jens Lehmann, Dhananjay Bhandiwad, Preetam Gattogi, Sahar Vahdati

MinPrompt: Graph-based minimal prompt data augmentation for few-shot question answering
Xiusi Chen, Jyun-Yu Jiang, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Wei Wang

Synthesizing conversations from unlabeled documents using automatic response segmentation
Fanyou Wu, Weijie Xu, Chandan Reddy, Srinivasan Sengamedu, "SHS"

Reasoning

Eliciting better multilingual structured reasoning from LLMs through code
Bryan Li, Tamer Alkhouli, Daniele Bonadiman, Nikolaos Pappas, Saab Mansour

II-MMR: Identifying and improving multi-modal multi-hop reasoning in visual question answering*
Jihyung Kil, Farideh Tavazoee, Dongyeop Kang, Joo-Kyung Kim

Recommender systems

Generative explore-exploit: Training-free optimization of generative recommender systems using LLM optimizers
Besnik Fetahu, Zhiyu Chen, Davis Yoshida, Giuseppe Castellucci, Nikhita Vedula, Jason Choi, Shervin Malmasi

Towards translating objective product attributes into customer language
Ram Yazdi, Oren Kalinsky, Alexander Libov, Dafna Shahaf

Responsible AI

SpeechGuard: Exploring the adversarial robustness of multimodal large language models
Raghuveer Peri, Sai Muralidhar Jayanthi, Srikanth Ronanki, Anshu Bhatia, Karel Mundnich, Saket Dingliwal, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Srikanth Vishnubhotla, Daniel Garcia-Romero, Sundararajan Srinivasan, Kyu Han, Katrin Kirchhoff

Text completion

Token alignment via character matching for subword completion*
Ben Athiwaratkun, Shiqi Wang, Mingyue Shang, Yuchen Tian, Zijian Wang, Sujan Gonugondla, Sanjay Krishna Gouda, Rob Kwiatkowski, Ramesh Nallapati, Bing Xiang

Token alignment.png
An illustration of token alignment process presented in "Token alignment via character matching for subword completion".

Research areas
Tags
About the Author
Staff writer

Related content

Work with us

See more jobs See more jobs
Applied Scientist Manager, Marketplace Intelligence
US, VA, Arlington
The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through SOTA generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. The Marketplace Intelligence (MI) team is looking for an Applied Science Manager to lead a team of scientists and engineers in building production ML and bandit solutions to customize the search experience. We determine which ads to show in Amazon search, where to place them, how many ads to place, and to which customers. This helps shoppers discover new products while helping advertisers put their products in front of the right customers, aligning shoppers’, advertisers’, and Amazon’s interests. To do this, we apply a broad range of machine learning, causal inference, and optimization techniques to continuously explore, learn, and optimize the allocation and ranking of ads on the search page. We are an interdisciplinary team with a focus on customer obsession and inventing and simplifying. Our primary focus is on improving the SP experience in search by gaining a deep understanding of shopper pain points and developing new innovative solutions to address them. You’ll lead the MI Interleaving team. The Interleaving team’s mission is to personalize and contextualize SP ad allocation on the search page. We do this by modeling shopper responses to the number, placement, and quality of ads. We use online experimentation, simulation, causal modeling, and online feedback to estimate the cost of displacing organic and sponsored content. Then, we incorporate those estimates into ad allocation to deliver an efficient and customized shopping experience for shoppers and improved discoverability and sales for advertisers. You’ll own the experimentation systems, models, and online model serving infrastructure to support these solutions. This is a unique opportunity for someone who wants to have broad business impact, a direct impact on customers and the search experience, build scaled real-time LLM and ML solutions, and lead a cross-functional team. If you are interested in machine learning, bandit learning, building production systems, and leading a team to build these solutions, this role is for you. We’re looking for a leader who can help lay out the vision for the team and grow with it. Key job responsibilities * Lead a team of scientists and engineers in building scalable machine learning solutions. * Develop a vision for contextualizing and personalizing SP ads in Amazon search. * Create, develop, and drive a data-driven product strategy to define the right quantitative measures of shopper impact, using this to evaluate decisions and opportunities. * Tackle and solve challenging science and business problems that balance the interests of advertisers, shoppers, and Amazon. * Own a portfolio of pragmatic long-term investments that drive long-term growth of the ads and retail businesses. * Develop real-time LLM and ML algorithms to allocate billions of ads per day in advertising auctions. * Develop efficient algorithms for multi-objective optimization and AI control methods to find operating points for the ad marketplace then evolve them * Develop scientists and ML engineers around machine learning, economics, and optimization for Advertising.
Data Scientist II - AMZ10564056
US, CA, Culver City
MULTIPLE POSITIONS AVAILABLE Employer: AMAZON.COM SERVICES LLC Offered Position: Data Scientist II Job Location: Culver City, California Job Number: AMZ10564056 Position Responsibilities: Design and implement scalable and reliable approaches to support or automate decision making throughout the business. Apply a range of data science techniques and tools combined with subject matter expertise to solve difficult business problems and cases in which the solution approach is unclear. Acquire data by building the necessary SQL / ETL queries. Import processes through various company specific interfaces for accessing Oracle, RedShift, and Spark storage systems. Build relationships with stakeholders and counterparts. Analyze data for trends and input validity by inspecting univariate distributions, exploring bivariate relationships, constructing appropriate transformations, and tracking down the source and meaning of anomalies. Build models using statistical modeling, mathematical modeling, econometric modeling, network modeling, social network modeling, natural language processing, machine learning algorithms, genetic algorithms, and neural networks. Validate models against alternative approaches, expected and observed outcome, and other business defined key performance indicators. Implement models that comply with evaluations of the computational demands, accuracy, and reliability of the relevant ETL processes at various stages of production. 40 hours / week, 8:00am-5:00pm, Salary Range: $158,681/year to $184,000/year. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, visit: https://www.aboutamazon.com/workplace/employee-benefits. Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation.#0000
AI Engineer - VLA model, RIVR
CH, Zurich
RIVR, an Amazon company, is building Physical AI by deploying autonomous robots for real-world doorstep delivery. Operating daily in diverse urban environments, RIVR's robots continuously learn from and navigate the millions of scenarios encountered during deliveries. By owning the full stack from software. Our fleet of delivery robots operates globally today, generating vast amounts of robotic real-world data. By utilizing state-of-the-art Vision-Language-Action (VLA) models, large-scale generalist models (like Transformers), generative AI, and similar methods, we can leverage this pool of data to significantly enhance its autonomy, navigation, and manipulation skills. In this role, you will develop multi-modal models that enable robots to autonomously generate actions from demonstrations, real-time sensor data, and natural language commands. We are seeking an expert in VLA models, imitation learning, and generative AI techniques with a deep knowledge of supervised, and self-supervised learning algorithms. If you are passionate about pushing the boundaries of AI we invite you to join us in shaping the future of intelligent robotics. Key job responsibilities Develop and implement Vision-Language-Action (VLA) models, generalist robot transformers, and imitation learning algorithms (e.g., diffusion policies) to enable robots to autonomously execute complex tasks. Design, test, and refine your algorithms to meet the demands of complex real-world autonomy and navigation tasks, with a focus on spatial reasoning and generalization. Streamline the data collection and training workflow to efficiently expand model capabilities with new tasks and data sources. Collaborate with the reinforcement learning team to innovate methods that leverage both simulated and real-world data. Optimize and distill networks for real-time deployment on the edge (e.g. Nvidia Jetson Thor). Build, lead and mentor an exceptional team of software engineers. Provide expert guidance to product managers and executives for strategic decision-making. Create and maintain documentation, guidelines, and best practices to streamline knowledge sharing.
Applied Scientist, EU INTech Consumer Selection Discovery, EU InTech Consumer Selection Discovery
ES, M, Madrid
At Amazon, we are committed to being the Earth’s most customer-centric company. The European International Technology group (EU INTech) owns the enhancement and delivery of Amazon’s engineering to all the varied customers and cultures of the world. We do this through a combination of partnerships with other Amazon technical teams and our own innovative new projects. You will be joining the Tools and Machine Learning (Tamale) team. As part of EU INTech, we tackle complex, high-impact problems at the intersection of e-commerce, AI, and customer experience; going well beyond traditional ML pipelines. Our work spans multiple frontiers: we build GenAI-powered data solutions, enabling scalable and cost-efficient training data generation at Amazon scale. We design and deploy agentic systems for detecting and fixing quality issues across Amazon's retail surfaces. And we develop ranking and recommendation models for Haul shopping experiences. We are looking for a passionate, talented, and inventive Scientist with a strong machine learning background to help build industry-leading machine learning solutions. We strongly value your hard work and obsession to solve complex problems on behalf of Amazon customers. Key job responsibilities We look for applied scientists who possess a wide variety of skills. As the successful applicant for this role, you will with work closely with your business partners to identify opportunities for innovation. You will apply machine learning solutions to automate manual processes, to scale existing systems and to improve catalog data quality, to name just a few. You will work with business leaders, scientists, and product managers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed services. You will be part of team of scientists and engineers working on solving data quality issues at scale. You will be able to influence the scientific roadmap of the team, setting the standards for scientific excellence. You will be working with state-of-the-art models, including image to text, Multimodal LLMs and GenAI. Your work will improve the experience of millions of daily customers using Amazon in Europe and in other regions. You will have the chance to have great customer impact and continue growing in one of the most innovative companies in the world. You will learn a huge amount - and have a lot of fun - in the process!
Economist II, PXT Central Science
CL, Virtual
The Central Science Team within Amazon’s People Experience and Technology org (PXTCS) uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, well-being, and the value of work to Amazonians. We are an interdisciplinary team, which combines the talents of science and engineering to develop and deliver solutions that measurably achieve this goal. We are looking for a Economist II who is able to provide structure around complex business problems, hone those complex problems into specific, scientific questions, and test those questions to generate insights. The ideal candidate will work with various science, engineering, operations, and analytics teams to estimate models and algorithms on large scale data, design pilots and measure their impact, and transform successful prototypes into improved policies and programs at scale. They will lead teams of researchers to produce robust, objective research results and insights which can be communicated to a broad audience inside and outside of Amazon. The ideal candidate has a PhD in Economics and deep expertise in causal inference and applied econometrics. Experience with large-scale data, proficiency in statistical programming (Python), and familiarity with machine learning methods are a plus. To be successful in this role, you should be comfortable operating with ambiguity, able to independently scope and prioritize research agendas, skilled at influencing decisions through rigorous analysis, and comfortable with using AI tools.
Applied Scientist, Machine Learning Accelerator - Annapurna Labs
US, MA, Boston
Our team is involved with pre-silicon design verification for custom IP. A critical requirement of the verification flow is the requirement of legal and realistic stimulus of a custom Machine Learning Accelerator Chip. Content creation is built using formal methods that model legal behavior of the design and then solving the problem to create the specific assembly tests. The entire frame work for creating these custom tests is developed using a SMT solver and custom software code to guide the solution space into templated scenarios. This highly visible and innovative role requires the design of this solving framework and collaborating with design verification engineers, hardware architects and designers to ensure that interesting content can be created for the projects needs. Key job responsibilities Develop an understanding for a custom machine learning instruction set architecture. Model correctness of instruction streams using first order logic. Create custom API's to allow control over scheduling and randomness. Deploy algorithms to ensure concurrent code is safely constructed. Create coverage metrics to ensure solution space coverage. Use novel methods like machine learning to automate content creation. About the team Utility Computing (UC) AWS Utility Computing (UC) provides product innovations — from foundational services such as Amazon’s Simple Storage Service (S3) and Amazon Elastic Compute Cloud (EC2), to consistently released new product innovations that continue to set AWS’s services and features apart in the industry. As a member of the UC organization, you’ll support the development and management of Compute, Database, Storage, Internet of Things (Iot), Platform, and Productivity Apps services in AWS, including support for customers who require specialized security solutions for customers who require specialized security solutions for their cloud services. Annapurna Labs (our organization within AWS UC) designs silicon and software that accelerates innovation. Customers choose us to create cloud solutions that solve challenges that were unimaginable a short time ago—even yesterday. Our custom chips, accelerators, and software stacks enable us to take on technical challenges that have never been seen before, and deliver results that help our customers change the world.
Senior Research Science Manager, Last Mile Science and Analytics
IN, KA, Bengaluru
Have you ever wondered how that Amazon box with the smile arrives so quickly, where it came from, and how much it cost Amazon to deliver? The WW Amazon Logistics, Business Analytics team manages the delivery of tens of millions of products every week to Amazon's customers, achieving on-time delivery in a cost-effective manner. We are seeking an enthusiastic, customer-obsessed Research Science Manager(RSM) with strong analytical skills to join our team. This role is crucial in optimizing Amazon's vast delivery network and will have significant impact on the customer experience, particularly in the final phase of delivery. As a Research Science Manager, you will: 1. Address business challenges through building compelling cases and using data to influence change across the organization 2. Develop input and assumptions based on preexisting models to estimate costs and savings opportunities associated with varying levels of network growth and operations 3. Create metrics to measure business performance, identify root causes and trends, and prescribe action plans 4. Manage multiple high-impact projects simultaneously 5. Work with technology teams and product managers to develop new tools and systems supporting business growth 6. Communicate with and support various internal stakeholders and external audiences 7. Implement scheduling solutions, improve metrics, and develop scalable processes and tools Looking for minds with: Extensive experience in operations research and data-driven decision making Strong analytical and problem-solving skills Robust program management and research science skills Ability to work with a team and make independent decisions in ambiguous environments Customer-obsessed mindset with a focus on improving the Amazon delivery experience This role offers the autonomy to think strategically and make data-driven decisions from day one. Join us in shaping the future of e-commerce delivery and addressing the core challenges in our world-class operations space! Key job responsibilities Key job responsibilities 1. Advanced Modeling and Algorithm Development: Design and implement sophisticated machine learning models for logistics optimization Develop complex time series forecasting algorithms for demand prediction and resource allocation 2. AI and Machine Learning Integration: Architect and deploy AI-powered systems to enhance decision-making in logistics operations Implement deep learning techniques for image recognition in package sorting and handling Develop reinforcement learning algorithms for adaptive scheduling and resource management 3. Big Data Analytics and Processing: Design and implement distributed computing solutions for processing massive logistics datasets Utilize cloud computing platforms (e.g., AWS) for scalable data processing and analysis 4. AI-Driven Workflow Optimization: Design and implement AI agents for autonomous decision-making in logistics processes Create machine learning models for customer behavior analysis and personalized delivery options 5. Software Development and System Architecture: Write efficient, scalable code in languages such as Python, Java, or C++ Develop and maintain complex software systems for logistics optimization Stay at the forefront of AI and ML research Publish research findings in top-tier conferences and journals About the team We are Amazon's Last Mile Science and Analytics team, dedicated to improving e-commerce delivery. We work to optimize our vast network, forecast demand using machine learning, and enhance route efficiency. Our efforts focus on developing innovative delivery methods, applying AI to solve complex problems, and conducting geospatial analysis. We create simulations to refine processes and plan capacity effectively. Operating globally, we strive to develop adaptable solutions for diverse markets. We aim to advance logistics science, continually improving speed, efficiency, and customer satisfaction, in support of Amazon's mission to be Earth's most customer-centric company.
Applied Scientist, SSG Science
IN, KA, Bengaluru
Amazon Devices is an inventive research and development company that designs and engineer high-profile devices like the Kindle family of products, Fire Tablets, Fire TV, Health Wellness, Amazon Echo & Astro products. This is an exciting opportunity to join Amazon in developing state-of-the-art techniques that bring Gen AI on edge for our consumer products. We are looking for exceptional scientists to join our Applied Science team and help develop the next generation of edge models, and optimize them while doing co-designed with custom ML HW based on a revolutionary architecture. Work hard. Have Fun. Make History. Key job responsibilities - Quantize, prune, distill, finetune Gen AI models to optimize for edge platforms - Fundamentally understand Amazon’s underlying Neural Edge Engine to invent optimization techniques - Analyze deep learning workloads and provide guidance to map them to Amazon’s Neural Edge Engine - Use first principles of Information Theory, Scientific Computing, Deep Learning Theory, Non Equilibrium Thermodynamics - Train custom Gen AI models that beat SOTA and paves path for developing production models - Collaborate closely with compiler engineers, fellow Applied Scientists, Hardware Architects and product teams to build the best ML-centric solutions for our devices - Publish in open source and present on Amazon's behalf at key ML conferences - NeurIPS, ICLR, MLSys.
Applied Scientist III - AMZ10564141
US, CA, Culver City
MULTIPLE POSITIONS AVAILABLE Employer: AMAZON.COM SERVICES LLC Offered Position: Applied Scientist III Job Location: Culver City, California Job Number: AMZ10564141 Position Responsibilities: Participate in the design, development, evaluation, deployment and updating of data-driven models and analytical solutions for machine learning (ML) and/or natural language (NL) applications. Develop and/or apply statistical modeling techniques (e.g. Bayesian models and deep neural networks), optimization methods, and other ML techniques to different applications in business and engineering. Routinely build and deploy ML models on available data, and run and analyze experiments in a production environment. Identify new opportunities for research in order to meet business goals. Research and implement novel ML and statistical approaches to add value to the business. Mentor junior engineers and scientists. 40 hours / week, 8:00am-5:00pm, Salary Range: $167,100/year to $226,100/year. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, visit: https://www.aboutamazon.com/workplace/employee-benefits. Amazon.com is an Equal Opportunity-Affirmative Action Employer – Minority / Female / Disability / Veteran / Gender Identity / Sexual Orientation.#0000
Applied Scientist II, Grocery, Retail & In-Store Experience (GRAISE)
IN, KA, Bengaluru
We are looking for a talented Applied Scientist to join our team. In this role, you will design, develop, and deploy machine learning and computer vision models that solve real-world problems at scale in the Amazon grocery domain. You will work closely with engineering, product, and business teams to turn complex technical challenges into production-ready solutions, and own the model development lifecycle from experimentation through deployment. You will bring scientific rigor to every stage — from data analysis and model design to evaluation and iteration. This is a high-impact role where your models will directly improve the shopping experience for millions of customers in Amazon grocery stores. Key job responsibilities Design, train, and evaluate computer vision and machine learning models for complex grocery-domain problems including product identification, shelf perception, and in-store scene understanding — iterating rapidly from prototype to production-quality solutions Conduct rigorous exploratory data analysis to characterize domain-specific challenges (image variability, catalog gaps, label noise) and translate findings into actionable modeling decisions Own the model development lifecycle from experimentation through deployment — collaborating with software and ML engineers to ensure models meet latency, throughput, and reliability requirements at production scale Design and execute offline and online evaluation frameworks — defining metrics that capture both model performance and downstream business impact, and diagnosing failure modes to prioritize improvements Build and improve data pipelines and annotation workflows that feed model training, including active learning strategies to maximize label efficiency Communicate technical results, trade-offs, and recommendations clearly to engineering, product, and business stakeholders — connecting model behavior to customer experience outcomes Stay current with state-of-the-art research in computer vision, multimodal learning, and representation learning — evaluating and adapting promising techniques to team-specific problems Contribute to a culture of scientific rigor through reproducible experimentation, thorough documentation, peer code and design reviews, and raising the quality bar for the team A day in the life As an Applied Scientist on the GRAISE team, you'll spend your days analyzing model performance from overnight experiments, collaborating with engineers to deploy computer vision models to production, and prototyping new approaches using multimodal learning with store video and sensor data. You'll present findings to product and business stakeholders, translating technical results into actionable recommendations. Throughout the day, you'll balance rigorous scientific thinking with practical engineering constraints, knowing your work directly improves the shopping experience for millions of customers in Amazon grocery stores. About the team The GRAISE team (Grocery, Retail & In-Store Experience) within World Wide Grocery Store Tech (WWGST) builds foundational AI and machine learning systems that power Amazon's in-store grocery technologies. We develop domain-specific models that solve uniquely complex challenges in grocery — from smart shopping carts and inventory intelligence to personalization and store operations. Our mission is to create technology which makes grocery shopping more convenient, economical, personalized, and enjoyable for customers while empowering retailers with operational efficiency
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social