Social
Conversational AI

A quick guide to Amazon’s papers at ACL 2024

Work on large language models predominates, with a particular focus on model evaluation.

By Staff writer
August 12, 2024
5 min read
Share

Like the field of conversational AI in general, Amazon’s papers at this year’s meeting of the Association for Computational Linguistics (ACL) are dominated by work on large language models (LLMs). The properties that make LLMs’ outputs so extraordinary — such as their linguistic fluency and semantic coherence — are also notoriously difficult to quantify; as such, model evaluation has emerged as a particular area of focus. But Amazon’s papers explore a wide range of LLM-related topics, from applications such as code synthesis and automatic speech recognition to problems of LLM training and deployment, such as continual pretraining and hallucination mitigation. Papers accepted to the recently inaugurated Proceedings of the ACL are marked with asterisks.

Code synthesis

Fine-tuning language models for joint rewriting and completion of code with potential bugs
Dingmin Wang, Jinman Zhao, Hengzhi Pei, Samson Tan, Sheng Zha

Bug injection.png
Obtaining buggy partial code via bug injection. From “Fine-tuning language models for joint rewriting and completion of code with potential bugs”.

Continual pretraining

Efficient continual pre-training for building domain specific large language models*
Yong Xie, Karan Aggarwal, Aitzaz Ahmad

Data quality

A shocking amount of the web is machine translated: Insights from multi-way parallelism*
Brian Thompson, Mehak Dhaliwal, Peter Frisch, Tobias Domhan, Marcello Federico

Document summarization

The power of summary-source alignments
Ori Ernst, Ori Shapira, Aviv Slobodkin, Sharon Adar, Mohit Bansal, Jacob Goldberger, Ran Levy, Ido Dagan

Hallucination mitigation

Learning to generate answers with citations via factual consistency models
Rami Aly, Zhiqiang Tang, Samson Tan, George Karypis

Intent classification

Can your model tell a negation from an implicature? Unravelling challenges with intent encoders
Yuwei Zhang, Siffi Singh, Sailik Sengupta, Igor Shalyminov, Hwanjun Song, Hang Su, Saab Mansour

Irony recognition

MultiPICo: Multilingual perspectivist irony corpus
Silvia Casola, Simona Frenda, Soda Marem Lo, Erhan Sezerer, Antonio Uva, Valerio Basile, Cristina Bosco, Alessandro Pedrani, Chiara Rubagotti, Viviana Patti, Davide Bernardi

Knowledge grounding

Graph chain-of-thought: Augmenting large language models by reasoning on graphs
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, Jiawei Han

MATTER: Memory-augmented transformer using heterogeneous knowledge sources*
Dongkyu Lee, Chandana Satya Prakash, Jack G. M. FitzGerald, Jens Lehmann

Tree-of-traversals: A zero-shot reasoning algorithm for augmenting black-box language models with knowledge graphs
Elan Markowitz, Anil Ramakrishna, Jwala Dhamala, Ninareh Mehrabi, Charith Peris, Rahul Gupta, Kai-Wei Chang, Aram Galstyan

Tree of traversals.png
An example of how the tree-of-traversals method uses a knowledge graph interface for the query “What actor played in both Inception and Interstellar?” From "Tree-of-traversals: A zero-shot reasoning algorithm for augmenting black-box language models with knowledge graphs".

LLM decoding

BASS: Batched attention-optimized speculative sampling*
Haifeng Qian, Sujan Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Anoop Deoras

Machine translation

Impacts of misspelled queries on translation and product search
Greg Hanneman, Natawut Monaikul, Taichi Nakatani

The fine-tuning paradox: Boosting translation quality without sacrificing LLM abilities
David Stap, Eva Hasler, Bill Byrne, Christof Monz, Ke Tran

Model editing

Propagation and pitfalls: Reasoning-based assessment of knowledge editing through counterfactual tasks
Wenyue Hua, Jiang Guo, Marvin Dong, Henghui Zhu, Patrick Ng, Zhiguo Wang

ReCoE construction.png
Demonstration of the process used to construct data for the reasoning-based counterfactual-editing (ReCoE) dataset. Straight lines represent data sourced from existing datasets; dashed lines denote data derived from LLM generation; zigzag lines denote data obtained through the corruption of other data. From "Propagation and pitfalls: Reasoning-based assessment of knowledge editing through counterfactual tasks".

Model evaluation

Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models
Francesco Tonolini, Jordan Massiah, Nikolaos Aletras, Gabriella Kazai

ConSiDERS—the-human evaluation framework: Rethinking human evaluation for generative large language models
Aparna Elangovan, Ling Liu, Lei Xu, Sravan Bodapati, Dan Roth

Factual confidence of LLMs: On reliability and robustness of current estimators
Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, Lluís Marquez

Fine-tuned machine translation metrics struggle in unseen domains
Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, Brian Thompson

Measuring question answering difficulty for retrieval-augmented generation
Matteo Gabburo, Nicolaas Jedema, Siddhant Garg, Leonardo Ribeiro, Alessandro Moschitti

Model robustness

Extreme miscalibration and the illusion of adversarial robustness
Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, George Karypis

Multimodal models

CaMML: Context-aware multimodal learner for large models
Yixin Chen, Shuai Zhang, Boran Han, Tong He, Bo Li

CAMML.png
The CaMML framework, which consists of a retriever, a perceiver and a generator. After receiving user query q, the CaMML retriever identifies relevant multimodal contexts C from the data store. Then the CaMML perceiver seamlessly integrates data of various modalities, effectively encoding long-context information and injecting it into the CaMML generator. This enables the prediction of responses that are conditioned on both the context and the query. From "CaMML: Context-aware multimodal learner for large models".

Multi-modal retrieval for large language model based speech recognition
Jari Kolehmainen, Aditya Gourav, Prashanth Gurunath Shivakumar, Yi Gu, Ankur Gandhe, Ariya Rastrow, Grant Strimel, Ivan Bulyko

REFINESUMM: Self-refining MLLM for generating a multimodal summarization dataset
Vaidehi Patil, Leonardo Ribeiro, Mengwen Liu, Mohit Bansal, Markus Dreyer

Ordinal classification

Exploring ordinality in text classification: A comparative study of explicit and implicit techniques
Siva Rajesh Kasa, Aniket Goel, Sumegh Roychowdhury, Karan Gupta, Anish Bhanushali, Nikhil Pattisapu, Prasanna Srinivasa Murthy

Question answering

Beyond boundaries: A human-like approach for question answering over structured and unstructured information sources*
Jens Lehmann, Dhananjay Bhandiwad, Preetam Gattogi, Sahar Vahdati

MinPrompt: Graph-based minimal prompt data augmentation for few-shot question answering
Xiusi Chen, Jyun-Yu Jiang, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Wei Wang

Synthesizing conversations from unlabeled documents using automatic response segmentation
Fanyou Wu, Weijie Xu, Chandan Reddy, Srinivasan Sengamedu, "SHS"

Reasoning

Eliciting better multilingual structured reasoning from LLMs through code
Bryan Li, Tamer Alkhouli, Daniele Bonadiman, Nikolaos Pappas, Saab Mansour

II-MMR: Identifying and improving multi-modal multi-hop reasoning in visual question answering*
Jihyung Kil, Farideh Tavazoee, Dongyeop Kang, Joo-Kyung Kim

Recommender systems

Generative explore-exploit: Training-free optimization of generative recommender systems using LLM optimizers
Besnik Fetahu, Zhiyu Chen, Davis Yoshida, Giuseppe Castellucci, Nikhita Vedula, Jason Choi, Shervin Malmasi

Towards translating objective product attributes into customer language
Ram Yazdi, Oren Kalinsky, Alexander Libov, Dafna Shahaf

Responsible AI

SpeechGuard: Exploring the adversarial robustness of multimodal large language models
Raghuveer Peri, Sai Muralidhar Jayanthi, Srikanth Ronanki, Anshu Bhatia, Karel Mundnich, Saket Dingliwal, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Srikanth Vishnubhotla, Daniel Garcia-Romero, Sundararajan Srinivasan, Kyu Han, Katrin Kirchhoff

Text completion

Token alignment via character matching for subword completion*
Ben Athiwaratkun, Shiqi Wang, Mingyue Shang, Yuchen Tian, Zijian Wang, Sujan Gonugondla, Sanjay Krishna Gouda, Rob Kwiatkowski, Ramesh Nallapati, Bing Xiang

Token alignment.png
An illustration of token alignment process presented in "Token alignment via character matching for subword completion".

Research areas
Tags
About the Author
Staff writer

Related content

Work with us

See more jobs See more jobs
Data Scientist
US, CA, San Francisco
If you are interested in this position, please apply on Twitch's Career site https://www.twitch.tv/jobs/en/ About Us: Twitch is the world’s biggest live streaming service, with global communities built around gaming, entertainment, music, sports, cooking, and more. It is where thousands of communities come together for whatever, every day. We’re about community, inside and out. You’ll find coworkers who are eager to team up, collaborate, and smash (or elegantly solve) problems together. We’re on a quest to empower live communities, so if this sounds good to you, see what we’re up to on LinkedIn and X, and discover the projects we’re solving on our Blog. Be sure to explore our Interviewing Guide to learn how to ace our interview process. About the Role We are looking for an experienced Data Scientist to support our central analytics and finance disciplines at Twitch. Bringing to bear a mixture of data analysis, dashboarding, and SQL query skills, you will use data-driven methods to answer business questions, and deliver insights that deepen understanding of our viewer behavior and monetization performance. Reporting to the VP of Finance, Analytics, and Business Operations, your team will be located in San Francisco. Our team is based in San Francisco, CA. You Will - Create actionable insights from data related to Twitch viewers, creators, advertising revenue, commerce revenue, and content deals. - Develop dashboards and visualizations to communicate points of view that inform business decision-making. - Create and maintain complex queries and data pipelines for ad-hoc analyses. - Author narratives and documentation that support conclusions. - Collaborate effectively with business partners, product managers, and data team members to align data science efforts with strategic goals. Perks * Medical, Dental, Vision & Disability Insurance * 401(k) * Maternity & Parental Leave * Flexible PTO * Amazon Employee Discount
Sr Applied Scientist, Amazon Shipping
IN, HR, Gurugram
Lead ML teams building large-scale forecasting and optimization systems that power Amazon’s global transportation network and directly impact customer experience and cost. As an Sr Applied Scientist, you will set scientific direction, mentor applied scientists, and partner with engineering and product leaders to deliver production-grade ML solutions at massive scale. Key job responsibilities 1. Lead and grow a high-performing team of Applied Scientists, providing technical guidance, mentorship, and career development. 2. Define and own the scientific vision and roadmap for ML solutions powering large-scale transportation planning and execution. 3. Guide model and system design across a range of techniques, including tree-based models, deep learning (LSTMs, transformers), LLMs, and reinforcement learning. 4. Ensure models are production-ready, scalable, and robust through close partnership with stakeholders. Partner with Product, Operations, and Engineering leaders to enable proactive decision-making and corrective actions. 5. Own end-to-end business metrics, directly influencing customer experience, cost optimization, and network reliability. 6. Help contribute to the broader ML community through publications, conference submissions, and internal knowledge sharing. A day in the life Your day includes reviewing model performance and business metrics, guiding technical design and experimentation, mentoring scientists, and driving roadmap execution. You’ll balance near-term delivery with long-term innovation while ensuring solutions are robust, interpretable, and scalable. Ultimately, your work helps improve delivery reliability, reduce costs, and enhance the customer experience at massive scale.
Senior Data Scientist, Pricing, Amazon Shipping, Amazon Shipping
US, WA, Bellevue
Who are we? Do you want to build Amazon's next $100B business? We're not just joining the shipping industry—we're transforming how billions of packages move across the world every year. Through evolving Amazon's controlled, predictable fulfillment network into a dynamic, adaptive shipping powerhouse we are building an intelligent system that optimizes in real-time to deliver on the promises businesses make to their customers. Our mission goes beyond moving boxes—we're spinning a flywheel where every new package makes our network stronger, faster, and more efficient. As we increase density and scale, we're revolutionizing shipping for businesses while simultaneously strengthening Amazon's own delivery capabilities, driving down costs and increasing speed for our entire ecosystem. What will you do? Amazon shipping is seeking a Senior Data Scientist with strong pricing and machine learning skills to work in an embedded team, partnering closely with commercial, product and tech. This person will be responsible for developing demand prediction models for Amazon shipping’s spot pricing system. As a Senior Data Scientist, you will be part of a science team responsible for improving price discovery across Amazon shipping, measuring the impact of model implementation, and defining a roadmap for improvements and expansion of the models into new unique use cases. This person will be collaborating closely with business and software teams to research, innovate, and solve high impact economics problems facing the worldwide Amazon shipping business. Who are you? The ideal candidate is analytical, resourceful, curious and team oriented, with clear communication skills and the ability to build strong relationships with key stakeholders. You should be a strong owner, are right a lot, and have a proven track record of taking on end-to-end ownership of and successfully delivering complex projects in a fast-paced and dynamic business environment. As this position involves regular interaction with senior leadership (director+), you need to be comfortable communicating at that level while also working directly with various functional teams. Key job responsibilities * Combine ML methodologies with fundamental economics principles to create new pricing algorithms. * Automate price exploration through automated experimentation methodologies, for example using multi-armed bandit strategies. * Partner with other scientists to dynamically predict prices to maximize capacity utilization. * Collaborate with product managers, data scientists, and software developers to incorporate models into production processes and influence senior leaders. * Educate non-technical business leaders on complex modeling concepts, and explain modeling results, implications, and performance in an accessible manner. * Independently identify and pursue new opportunities to leverage economic insights * Opportunity to expand into other domains such as causal analytics, optimization and simulation. About the team Amazon Shipping's pricing team empowers our global business to find strategic harmony between growth and profit tradeoffs, while seeking long term customer value and financial viability. Our people and systems help identify and drive synergy between demand, operational, and economic planning. The breadth of our problems range from CEO-level strategic support to in-depth mathematical experimentation and optimization. Excited by the intersection of data and large scale strategic decision-making? This is the team for you!
Economist, Prime Science, Prime Science
US, WA, Seattle
Amazon Prime is looking for an ambitious Economist to help create econometric insights for world-wide Prime. Prime is Amazon's premiere membership program, with over 200M members world-wide. This role is at the center of many major company decisions that impact Amazon's customers. These decisions span a variety of industries, each reflecting the diversity of Prime benefits. These range from fast-free e-commerce shipping, digital content (e.g., exclusive streaming video, music, gaming, photos), reading, healthcare, and grocery offerings. Prime Science creates insights that power these decisions. As an economist in this role, you will create statistical tools that embed causal interpretations. You will utilize massive data, state-of-the-art scientific computing, econometrics (causal, counterfactual/structural, experimentation), and machine-learning, to do so. Some of the science you create will be publishable in internal or external scientific journals and conferences. You will work closely with a team of economists, applied scientists, data professionals (business analysts, business intelligence engineers), product managers, and software/data engineers. You will create insights from descriptive statistics, as well as from novel statistical and econometric models. You will create internal-to-Amazon-facing automated scientific data products to power company decisions. You will write strategic documents explaining how senior company leaders should utilize these insights to create sustainable value for customers. These leaders will often include the senior-most leaders at Amazon. The team is unique in its exposure to company-wide strategies as well as senior leadership. It operates at the research frontier of utilizing data, econometrics, artificial intelligence, and machine-learning to form business strategies. A successful candidate will have demonstrated a capacity for building, estimating, and defending statistical models (e.g., causal, counterfactual, machine-learning) using software such as R, Python, or STATA. They will have a willingness to learn and apply a broad set of statistical and computational techniques to supplement deep training in one area of econometrics. For example, many applications on the team motivate the use of structural econometrics and machine-learning. They rely on building scalable production software, which involves a broad set of world-class software-building skills often learned on-the-job. As a consequence, already-obtained knowledge of SQL, machine learning, and large-scale scientific computing using distributed computing infrastructures such as Spark-Scala or PySpark would be a plus. Additionally, this candidate will show a track-record of delivering projects well and on-time, preferably in collaboration with other team members (e.g. co-authors). Candidates must have very strong writing and emotional intelligence skills (for collaborative teamwork, often with colleagues in different functional roles), a growth mindset, and a capacity for dealing with a high-level of ambiguity. Endowed with these traits and on-the-job-growth, the role will provide the opportunity to have a large strategic, world-wide impact on the customer experiences of Prime members.
Data Scientist- Vision Language Model Researcher, Data & Machine Learning (DML)
US, VA, Arlington
This position requires that the candidate selected be a US Citizen and currently possess and maintain an active Top Secret security clearance. Join a sizeable team of data scientists, research scientists, and machine learning engineers that develop vision language models (VLMs) on overhead imagery for a high-impact government customer. We own the entire machine learning development life cycle, developing models on customer data: - Exploring the data and brainstorming and prioritizing ideas for model development - Implementing new features - Training models in support of experimental or performance goals - T&E-ing, packaging, and delivering models We perform this work on both unclassified and classified networks, with portions of our team working on each network. We seek a new team member to work on the classified networks. You would work collaboratively with teammates to develop and use a python codebase for fine-tuning VLMs. You would have great opportunities to learn from team members and technical leads, while also having opportunities for ownership of important project workflows. You would work with Jupyter Notebooks, the Linux command line, GitLab, and Visual Studio Code. Key job responsibilities With support from technical leads, carry out tasking across the entire machine learning development lifecycle to fine-tune VLMs on overhead imagery: - Run data conversion pipelines to transform customer data into the structure needed by models for training - Perform EDA on the customer data - Train VLMs on overhead imagery - Develop and implement hyper-parameter optimization strategies - Test and Evaluate models and analyze results - Package and deliver models to the customer - Implement new features to the code base - Collaborate with the rest of the team on long term strategy and short-medium term implementation. - Contribute to presentations to the customer regarding the team’s work.
Data Scientist - Computer Vision, Data & Machine Learning (DML)
US, VA, Arlington
This position requires that the candidate selected be a US Citizen and currently possess and maintain an active Top Secret security clearance. Join a sizeable team of data scientists, research scientists, and machine learning engineers that develop computer vision models on overhead imagery for a high-impact government customer. We own the entire machine learning development life cycle, developing models on customer data: - Exploring the data and brainstorming and prioritizing ideas for model development - Implementing new features in our sizable code base - Training models in support of experimental or performance goals - T&E-ing, packaging, and delivering models We perform this work on both unclassified and classified networks, with portions of our team working on each network. We seek a new team member to work on the classified networks. Three to four days a week, you would travel to the customer site in Northern Virginia to perform tasking as described below. Weekdays when you do not travel to the customer site, you would work from your local Amazon office. You would work collaboratively with teammates to use and contribute to a well-maintained code base that the team has developed over the last several years, almost entirely in python. You would have great opportunities to learn from team members and technical leads, while also having opportunities for ownership of important project workflows. You would work with Jupyter Notebooks, the Linux command line, Apache AirFlow, GitLab, and Visual Studio Code. We are a very collaborative team, and regularly teach and learn from each other, so, if you are familiar with some of these technologies, but unfamiliar with others, we encourage you to apply - especially if you are someone who likes to learn. We are always learning on the job ourselves. Key job responsibilities With support from technical leads, carry out tasking across the entire machine learning development lifecycle to develop computer vision models on overhead imagery: - Run data conversion pipelines to transform customer data into the structure needed by models for training - Perform EDA on the customer data - Train deep neural network models on overhead imagery - Develop and implement hyper-parameter optimization strategies - Test and Evaluate models and analyze results - Package and deliver models to the customer - Incorporate model R&D from low-side researchers - Implement new features to the model development code base - Collaborate with the rest of the team on long term strategy and short-medium term implementation. - Contribute to presentations to the customer regarding the team’s work.
Sr. Applied Scientist - SLAM, Amazon Industrial Robotics
US, MA, N.reading
Amazon Industrial Robotics (AIR) is seeking exceptional talent to help develop the next generation of advanced robotics systems that will transform automation at Amazon's scale. We're building revolutionary robotic systems that combine cutting-edge AI, sophisticated control systems, and advanced mechanical design to create adaptable automation solutions capable of working safely alongside humans in dynamic environments. This is a unique opportunity to shape the future of robotics and automation at an unprecedented scale, working with world-class teams pushing the boundaries of what's possible in robotic dexterous manipulation, locomotion, and human-robot interaction. This role presents an opportunity to shape the future of robotics through innovative applications of the latest software and AI tools for robots. We are seeking an expert to lead the development of our SLAM and Spatial AI module. In this role, you will create methods that will enable our robot to perceive the environment and navigate with unrivaled vision and fidelity. The system will combine an array of diverse sensors with simultaneous localization and mapping software that continuously updates the map in real-time automatically. It will have the capability to ‘see’ and identify different objects, people, vehicles, and places as it moves and react to moving people and vehicles in an intelligent way. The system combines a mix of high-performance sensors with simultaneous localization and mapping software that builds and continuously updates maps in real-time, completely automatically. It has the capability to ‘see’ and identify different objects, people, vehicles, and places as it moves and react to moving people and vehicles in an intelligent way. Key job responsibilities - Analyze, design, develop, and test existing and new perception capabilities using cameras and LIDAR sensor inputs for obstacle detection and semantic understanding. - Research, design, implement and evaluate scientific approaches to a variety of autonomy challenges.. - Create experiments and prototype implementations of new perception algorithms. - Deliver high quality production level code (C++ or Python) and support systems in production. - Collaborate with other functional teams in a robotics organization. - Collaborate closely with hardware engineering team members on developing systems from prototyping to production level. - Represent Amazon in academia community through publications and scientific presentations. - Work with stakeholders across hardware, science, and operations teams to iterate on systems design and implementation.
Applied Scientist - Delivery Experience, DEX Speed & AI
US, WA, Bellevue
Why this job is awesome? - This is SUPER high-visibility work: Our mission is to provide consistent, accurate, and relevant delivery information to every single page on every Amazon-owned site. - MILLIONS of customers will be impacted by your contributions: The changes we make directly impact the customer experience on every Amazon site. This is a great position for someone who likes to leverage Machine learning technologies to solve the real customer problems, and also wants to see and measure their direct impact on customers. - We are a cross-functional team that owns the ENTIRE delivery experience for customers: From the business requirements to the technical systems that allow us to directly affect the on-site experience from a central service, business and technical team members are integrated so everyone is involved through the entire development process. - Do you want to join an innovative team of scientists and engineers who use optimization, machine learning and Gen-AI techniques to deliver the best delivery experience on every Amazon-owned site? - Are you excited by the prospect of analyzing and modeling terabytes of data on the cloud and create state-of-art algorithms to solve real world problems? - Do you like to own end-to-end business problems/metrics and directly impact the same-day delivery service of Amazon? - Do you like to innovate and simplify? If yes, then you may be a great fit to join the Delivery Experience Machine Learning team! Key job responsibilities · Research and implement Optimization, ML and Gen-AI techniques to create scalable and effective models in Delivery Experience (DEX) systems · Design and develop optimization models and reinforcement learning models to improve quality of same-day selections · Apply LLM technology to empower CX features · Establishing scalable, efficient, automated processes for large scale data analysis and causal inference
Economist, People Experience and Technology Central Science
US, CA, San Francisco
The People eXperience and Technology Central Science (PXTCS) team uses economics, behavioral science, statistics, and machine learning to proactively identify mechanisms and process improvements which simultaneously improve Amazon and the lives, wellbeing, and the value of work to Amazonians. PXTCS is an interdisciplinary team that combines the talents of science and engineering to develop and deliver solutions that measurably achieve this goal. PXTCS is looking for an economist who can apply economic methods to address business problems. The ideal candidate will work with engineers and computer scientists to estimate models and algorithms on large scale data, design pilots and measure impact, and transform successful prototypes into improved policies and programs at scale. PXTCS is looking for creative thinkers who can combine a strong technical economic toolbox with a desire to learn from other disciplines, and who know how to execute and deliver on big ideas as part of an interdisciplinary technical team. Ideal candidates will work in a team setting with individuals from diverse disciplines and backgrounds. They will work with teammates to develop scientific models and conduct the data analysis, modeling, and experimentation that is necessary for estimating and validating models. They will work closely with engineering teams to develop scalable data resources to support rapid insights, and take successful models and findings into production as new products and services. They will be customer-centric and will communicate scientific approaches and findings to business leaders, listening to and incorporate their feedback, and delivering successful scientific solutions. A day in the life The Economist will work with teammates to apply economic methods to business problems. This might include identifying the appropriate research questions, writing code to implement a DID analysis or estimate a structural model, or writing and presenting a document with findings to business leaders. Our economists also collaborate with partner teams throughout the process, from understanding their challenges, to developing a research agenda that will address those challenges, to help them implement solutions. About the team PXTCS is a multidisciplinary science team that develops innovative solutions to make Amazon Earth's Best Employer
Sr. Applied Scientist, Sponsored Products Off-Search Sourcing
US, NY, New York
The Sponsored Products and Brands (SPB) team at Amazon Ads is re-imagining the advertising landscape through state-of-the-art generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. The Off-Search team within Sponsored Products and Brands (SPB) is focused on building delightful ad experiences across various surfaces beyond Search on Amazon—such as product detail pages, the homepage, and store-in-store pages—to drive monetization. Our vision is to deliver highly personalized, context-aware advertising that adapts to individual shopper preferences, scales across diverse page types, remains relevant to seasonal and event-driven moments, and integrates seamlessly with organic recommendations such as new arrivals, basket-building content, and fast-delivery options. To execute this vision, we work in close partnership with Amazon Stores stakeholders to lead the expansion and growth of advertising across Amazon-owned and -operated pages beyond Search. We operate full stack—from backend ads-retail edge services, ads retrieval, and ad auctions to shopper-facing experiences—all designed to deliver meaningful value. Curious about our advertising solutions? Discover more about Sponsored Products and Sponsored Brands to see how we’re helping businesses grow on Amazon.com and beyond! Key job responsibilities This role will be pivotal in redesigning how ads contribute to a personalized, relevant, and inspirational shopping experience, with the customer value proposition at the forefront. Key responsibilities include, but are not limited to: - Contribute to the design and development of GenAI, deep learning, multi-objective optimization and/or reinforcement learning empowered solutions to transform ad retrieval, auctions, whole-page relevance, and/or bespoke shopping experiences. - Collaborate cross-functionally with other scientists, engineers, and product managers to bring scalable, production-ready science solutions to life. - Stay abreast of industry trends in GenAI, LLMs, and related disciplines, bringing fresh and innovative concepts, ideas, and prototypes to the organization. - Contribute to the enhancement of team’s scientific and technical rigor by identifying and implementing best-in-class algorithms, methodologies, and infrastructure that enable rapid experimentation and scaling. - Mentor and grow junior scientists and engineers, cultivating a high-performing, collaborative, and intellectually curious team. A day in the life As an Applied Scientist on the Sponsored Products and Brands Off-Search team, you will contribute to the development in Generative AI (GenAI) and Large Language Models (LLMs) to revolutionize our advertising flow, backend optimization, and frontend shopping experiences. This is a rare opportunity to redefine how ads are retrieved, allocated, and/or experienced—elevating them into personalized, contextually aware, and inspiring components of the customer journey. You will have the opportunity to fundamentally transform areas such as ad retrieval, ad allocation, whole-page relevance, and differentiated recommendations through the lens of GenAI. By building novel generative models grounded in both Amazon’s rich data and the world’s collective knowledge, your work will shape how customers engage with ads, discover products, and make purchasing decisions. If you are passionate about applying frontier AI to real-world problems with massive scale and impact, this is your opportunity to define the next chapter of advertising science. About the team The Off-Search team within Sponsored Products and Brands (SPB) is focused on building delightful ad experiences across various surfaces beyond Search on Amazon—such as product detail pages, the homepage, and store-in-store pages—to drive monetization. Our vision is to deliver highly personalized, context-aware advertising that adapts to individual shopper preferences, scales across diverse page types, remains relevant to seasonal and event-driven moments, and integrates seamlessly with organic recommendations such as new arrivals, basket-building content, and fast-delivery options. To execute this vision, we work in close partnership with Amazon Stores stakeholders to lead the expansion and growth of advertising across Amazon-owned and -operated pages beyond Search. We operate full stack—from backend ads-retail edge services, ads retrieval, and ad auctions to shopper-facing experiences—all designed to deliver meaningful value. Curious about our advertising solutions? Discover more about Sponsored Products and Sponsored Brands to see how we’re helping businesses grow on Amazon.com and beyond!
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our monthly newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social