New workshop to help bring causal reasoning to recommendation systems

Two-day RecSys workshop that extends the popular REVEAL to include CONSEQUENCES features Amazon organizers, speakers.

The ACM Conference on Recommender Systems (RecSys) is the premier conference in the field of recommender systems, with recent editions attracting well over a thousand participants each, from both academic and industry backgrounds. This diversity is something RecSys prides itself in, as it leads to valuable cross-pollination and fosters progress in interdisciplinary research.

Naturally, applications of recommender systems are an important research area for Amazon, as we aim to delight customers on our retail website, Amazon Music, and Prime Video and with Alexa and more. We are proud that this year the conference will be held in Seattle, with two Amazon scientists serving as general co-chairs and even more Amazon scientists on the organizing and program committees.

Consequences.16x9.png
Guido Imbens (left), the applied econometrics professor and professor of economics at the Stanford Graduate School of Business, a research consultant at Amazon, and a 2021 recipient of the Nobel Prize for economics, and Lihong Li, a senior principal scientist at Amazon, are the two invited speakers at the CONSEQUENCES+REVEAL workshop.

RecSys also has a track record of strong workshop programs, where researchers from industry and academia collaborate to dive deep on emerging research topics and open problems in the field. We are happy to contribute to this trend this year with the two-day CONSEQUENCES+REVEAL workshop.

The REVEAL workshop series ran from 2018 to 2020, addressing problems that ranged from offline evaluation to bandits and reinforcement learning, generating lots of interest. REVEAL will be reinstated this year, focusing on large-scale reinforcement learning for recommendation.

We’re also extending its scope with CONSEQUENCES: Causality, Counterfactuals, and Sequential Decision-Making for Recommender Systems. Traditionally, machine learning approaches to recommendation tend to cast it as a prediction task: “What is the probability that this user would like this product?”

Related content
In 2017, when the journal IEEE Internet Computing was celebrating its 20th anniversary, its editorial board decided to identify the single paper from its publication history that had best withstood the “test of time”. The honor went to a 2003 paper called “Amazon.com Recommendations: Item-to-Item Collaborative Filtering”, by then Amazon researchers Greg Linden, Brent Smith, and Jeremy York.

Recently, the community has begun to realize that this is a myopic way of looking at the problem, and that, rather than predictions, our systems should make decisions. This is an important distinction, as decisions have consequences, and the recommendations we provide can influence shoppers’ behavior, sellers’ exposure, future training data for our algorithms, and so on.

If we want to reason about the (possibly unintended) consequences of the decisions our machine learning models make, we need to borrow ideas from causal inference. Typical “what if”-type questions in learning and evaluation require a way to model counterfactuals. Accounting for causal factors can foster progress in effective, efficient and fair learning and evaluation from logged data.

CONSEQUENCES will host two invited speakers: a Nobel Prize–winning academic pioneer and an industrial research giant with over a decade of experience in both fundamental theory and applications of the topic at hand.

Guido Imbens

Guido cropped.png
Guido Imbens

Guido Imbens is the applied econometrics professor and professor of economics at the Stanford Graduate School of Business and a research consultant at Amazon. After graduating from Brown University, Guido taught at Harvard University, UCLA, and UC Berkeley. He joined Stanford in 2012. Imbens specializes in econometrics and, in particular, methods for drawing causal inferences. He is a fellow of the Econometric Society and the American Academy of Arts and Sciences. In 2021, he was co-awarded the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel for "methodological contributions to the analysis of causal relationships".

Lihong Li

Lihong cropped.png
Lihong Li

Lihong Li is a senior principal scientist at Amazon. He received a PhD in computer science from Rutgers University and has held research positions at Yahoo!, Microsoft, and Google. His main research interests are reinforcement learning, including contextual bandits, and related problems in AI. His work has found applications in recommendation, advertising, web search, and conversation systems, and he has won best-paper awards at ICML, AISTATS, and WSDM. He regularly serves as area chair or senior program committee member at major AI/ML conferences such as AAAI, AISTATS, ICLR, ICML, IJCAI, and NeurIPS.

The organizers of CONSEQUENCES+REVEAL are working together to implement a novel workshop format filled with exciting content, where the program will be divided over two days, based on topic. We will include a poster session for all accepted contributions, oral presentations for selected contributions, and an in-workshop tutorial at the start to introduce advanced concepts and techniques.
See the official workshop website for more information on how to submit your contributions. We look forward to seeing you there!

Important dates

Submission deadline

August 5, 2022

Author notification

August 27, 2022

Camera-ready-version deadline

September 10, 2022

CONSEQUENCES ’22

September 18–23, 2022

REVEAL ’22

September 18-23, 2022

Organizers

Related content
Amazon Scholar David Card and Amazon academic research consultant Guido Imbens talk about the past and future of empirical economics.

CONSEQUENCES

REVEAL

Related content

US, WA, Seattle
We are looking for a Principal Applied Scientist to drive the research and development of real-time multimodal conversational AI. You will operate across two focus areas: advancing foundation models for speech and audio, and building the post-training systems (reward modeling, reinforcement learning) that shape natural, human-like conversational behavior. You will be the expert in your area while contributing across the full model lifecycle — from pre-training and architecture design through post-training alignment and real-time deployment. You will work at the frontier of what's possible in conversational AI, with the compute, data, and runway to pursue problems that few teams in the world have the resources to tackle. As a Principal Scientist, you will set the technical direction for your research area, influence the broader roadmap, and work closely with inference engineers to ensure your models are designed for real-time production deployment from inception. Key job responsibilities Foundation Model Scaling - Build and train large-scale multimodal foundation models for real-time speech and audio generation, from architecture design through production-scale training - Advance the scaling and efficiency of conversational modes, including the relationship between data, model size, and real time performance. - Design model architectures informed by hardware constraints and inference requirements, working with inference engineers to ensure models are servable from inception - Develop training methodologies for multimodal models that jointly process and generate speech, language, and audio in real-time streaming contexts Post-Training & Reinforcement Learning - Design and build reward models and reward functions for speech systems — capturing naturalness, fluency, conversational quality, and real-time responsiveness - Develop and apply reinforcement learning methods to shape conversational behavior — teaching models natural timing, responsiveness, and fluid interaction - Build the post-training pipeline from SFT through RL alignment, optimized for real-time multimodal outputs rather than text-only generation - Design evaluation frameworks that capture the quality dimensions unique to real-time conversation Real-Time Perception & Generation - Advance the team's capabilities in real-time perception - Work at the intersection of model architecture and production constraints to ensure multimodal capabilities function within hard real-time latency budgets
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Member of Technical Staff with a strong deep learning background, to build industry-leading Generative Artificial Intelligence (GenAI) technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As a Member of Technical Staff with the AGI team, you will lead the development of algorithms and modeling techniques, to advance the state of the art with LLMs. You will lead the foundational model development in an applied research role, including model training, dataset design, and pre- and post-training optimization. Your work will directly impact our customers in the form of products and services that make use of GenAI technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in LLMs. About the team The AGI team has a mission to push the envelope in GenAI with LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
US, WA, Redmond
We are searching for a talented candidate with experience in orbital mechanics, orbit determination, launch vehicle trajectories, and launch vehicle mission planning. In this position, a successful candidate would serve as a Research Scientist in support of Amazon Leo’s constellation with particular focus on Launch Vehicle support. Strong analysis skills are required to develop engineering studies of complex large-scale dynamical systems. This position requires demonstrated expertise in computational analysis automation and tool development. Export Control Requirement: Due to applicable export control laws and regulations, candidates must be a U.S. citizen or national, U.S. permanent resident (i.e., current Green Card holder), or lawfully admitted into the U.S. as a refugee or granted asylum. Key job responsibilities Working with the Leo GNC team, you will: • Perform spacecraft maneuver or navigation analysis in support of multi-disciplinary trades within the Amazon Leo team. • Contribute to prototype software development of flight algorithms. • Test and assess navigation software for integration into flight systems. • Assess and trouble-shoot the performance of Leo on-board GNSS hardware and software systems. • Work closely with GNC engineers to manage on-orbit performance and develop flight dynamics operations processes. • Manage engineering trades as needed for various launch vehicle mission designs. • Support Leo’s Launch Vehicle Mission Management team with technical expertise in Launch Vehicle trajectory requirements specification • Evaluate Launch Vehicle performance and compliance with mission requirements • Develop tools to support Mission Management planning for over 80 launches! • Work collaboratively with launch vehicle system technical teams About the team The Flight Dynamics team is responsible for the guidance, navigation, control and safety of the spaceflight of the Amazon Leo constellation. This team provides solutions to spaceflight challenges in constellation design, orbit selection, launch vehicle insertion requirements, navigation, trajectory design, space situational awareness, and space traffic coordination.
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Member of Technical Staff with a strong deep learning background, to build industry-leading Generative Artificial Intelligence (GenAI) technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As a Member of Technical Staff with the AGI team, you will lead the development of algorithms and modeling techniques, to advance the state of the art with LLMs. You will lead the foundational model development in an applied research role, including model training, dataset design, and pre- and post-training optimization. Your work will directly impact our customers in the form of products and services that make use of GenAI technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in LLMs. About the team The AGI team has a mission to push the envelope in GenAI with LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Member of Technical Staff with a strong deep learning background, to build industry-leading Generative Artificial Intelligence (GenAI) technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As a Member of Technical Staff with the AGI team, you will lead the development of algorithms and modeling techniques, to advance the state of the art with LLMs. You will lead the foundational model development in an applied research role, including model training, dataset design, and pre- and post-training optimization. Your work will directly impact our customers in the form of products and services that make use of GenAI technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in LLMs. About the team The AGI team has a mission to push the envelope in GenAI with LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Member of Technical Staff with a strong deep learning background, to build industry-leading Generative Artificial Intelligence (GenAI) technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As a Member of Technical Staff with the AGI team, you will lead the development of algorithms and modeling techniques, to advance the state of the art with LLMs. You will lead the foundational model development in an applied research role, including model training, dataset design, and pre- and post-training optimization. Your work will directly impact our customers in the form of products and services that make use of GenAI technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in LLMs. About the team The AGI team has a mission to push the envelope in GenAI with LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Member of Technical Staff with a strong deep learning background, to build industry-leading Generative Artificial Intelligence (GenAI) technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As a Member of Technical Staff with the AGI team, you will lead the development of algorithms and modeling techniques, to advance the state of the art with LLMs. You will lead the foundational model development in an applied research role, including model training, dataset design, and pre- and post-training optimization. Your work will directly impact our customers in the form of products and services that make use of GenAI technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in LLMs. About the team The AGI team has a mission to push the envelope in GenAI with LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
IN, KA, Bengaluru
Amazon is looking for a passionate, talented, and inventive Data Scientist with machine learning background to help build industry-leading Speech and Language technology. Our mission is to provide a delightful experience to Amazon’s customers by pushing the envelope in Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), Machine Learning (ML). Key job responsibilities Key job responsibilities Amazon is looking for a passionate, talented, and inventive Data Scientist with machine learning background to help build industry-leading Speech and Language technology. Our mission is to provide a delightful experience to Amazon’s customers by pushing the envelope in Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), Machine Learning (ML) and Computer Vision (CV). As part of our AI team in Amazon AWS, you will work alongside internationally recognized experts to develop data experiments, novel algorithms and data techniques to advance the state-of-the-art in human language technology. Your work will directly impact millions of our customers in the form of products and services that make use of speech and language technology. You will gain hands on experience with Amazon’s heterogeneous speech, text, and structured data sources, and large-scale computing resources to accelerate advances in spoken language understanding.
US, WA, Seattle
We are looking for detail-oriented, organized, and responsible individuals who are eager to learn how to apply their causal inference / structural econometrics skillsets to solve real world problems. The intern will work in the area of Store Economics and Science (SEAS) and develop models to SEAS. Our PhD Economist Internship Program offers hands-on experience in applied economics, supported by mentorship, structured feedback, and professional development. Interns work on real business and research problems, building skills that prepare them for full-time economist roles at Amazon and beyond. You will learn how to build data sets and perform applied econometric analysis collaborating with economists, scientists, and product managers. These skills will translate well into writing applied chapters in your dissertation and provide you with work experience that may help you with placement. About the team The Stores Economics and Science Team (SEAS) is a Stores-wide interdisciplinary team at Amazon with a "peak jumping" mission focused on disruptive innovation. The team applies science, economics, and engineering expertise to tackle the business's most critical problems, working to move from local to global optima across Amazon Stores operations. SEAS builds partnerships with organizations throughout Amazon Stores to pursue this mission, exploring frontier science while learning from the experience and perspective of others. Their approach involves testing solutions first at a small scale, then aligning more broadly to build scalable solutions that can be implemented across the organization. The team works backwards from customers using their unique scientific expertise to add value, takes on long-run and high-risk projects that business teams typically wouldn't pursue, helps teams with kickstart problems by building practical prototypes, raises the scientific bar at Amazon, and builds and shares software that makes Amazon more productive.
US, WA, Seattle
Prime Video is a first-stop entertainment destination offering customers a vast collection of premium programming in one app available across thousands of devices. Prime members can customize their viewing experience and find their favorite movies, series, documentaries, and live sports – including Amazon MGM Studios-produced series and movies; licensed fan favorites; and programming from Prime Video add-on subscriptions such as Apple TV+, Max, Crunchyroll and MGM+. All customers, regardless of whether they have a Prime membership or not, can rent or buy titles via the Prime Video Store, and can enjoy even more content for free with ads. Are you interested in shaping the future of entertainment? Prime Video's technology teams are creating best-in-class digital video experience. As a Prime Video technologist, you’ll have end-to-end ownership of the product, user experience, design, and technology required to deliver state-of-the-art experiences for our customers. You’ll get to work on projects that are fast-paced, challenging, and varied. You’ll also be able to experiment with new possibilities, take risks, and collaborate with remarkable people. We’ll look for you to bring your diverse perspectives, ideas, and skill-sets to make Prime Video even better for our customers. With global opportunities for talented technologists, you can decide where a career Prime Video Tech takes you! We are looking for a self-motivated, passionate and resourceful Applied Scientist to bring diverse perspectives, ideas, and skill-sets to make Prime Video even better for our customers. You will spend your time as a hands-on machine learning practitioner and a research leader. You will play a key role on the team, building and guiding machine learning models from the ground up. At the end of the day, you will have the reward of seeing your contributions benefit millions of Amazon.com customers worldwide. Key job responsibilities Develop AI solutions for various Prime Video Personalization systems using Deep learning, GenAI, Reinforcement Learning, and optimization methods; Work closely with engineers and product managers to design, implement and launch AI solutions end-to-end; Design and conduct offline and online (A/B) experiments to evaluate proposed solutions based on in-depth data analyses; Effectively communicate technical and non-technical ideas with teammates and stakeholders; Stay up-to-date with advancements and the latest modeling techniques in the field; Publish your research findings in top conferences and journals. About the team Prime Video Personalization and Discovery team owns science solution to power personalized experience on various devices, from sourcing, relevance, ranking, to name a few. We work closely with the engineering and product teams to launch our solutions in production.