New workshop to help bring causal reasoning to recommendation systems

Two-day RecSys workshop that extends the popular REVEAL to include CONSEQUENCES features Amazon organizers, speakers.

The ACM Conference on Recommender Systems (RecSys) is the premier conference in the field of recommender systems, with recent editions attracting well over a thousand participants each, from both academic and industry backgrounds. This diversity is something RecSys prides itself in, as it leads to valuable cross-pollination and fosters progress in interdisciplinary research.

Naturally, applications of recommender systems are an important research area for Amazon, as we aim to delight customers on our retail website, Amazon Music, and Prime Video and with Alexa and more. We are proud that this year the conference will be held in Seattle, with two Amazon scientists serving as general co-chairs and even more Amazon scientists on the organizing and program committees.

Consequences.16x9.png
Guido Imbens (left), the applied econometrics professor and professor of economics at the Stanford Graduate School of Business, a research consultant at Amazon, and a 2021 recipient of the Nobel Prize for economics, and Lihong Li, a senior principal scientist at Amazon, are the two invited speakers at the CONSEQUENCES+REVEAL workshop.

RecSys also has a track record of strong workshop programs, where researchers from industry and academia collaborate to dive deep on emerging research topics and open problems in the field. We are happy to contribute to this trend this year with the two-day CONSEQUENCES+REVEAL workshop.

The REVEAL workshop series ran from 2018 to 2020, addressing problems that ranged from offline evaluation to bandits and reinforcement learning, generating lots of interest. REVEAL will be reinstated this year, focusing on large-scale reinforcement learning for recommendation.

We’re also extending its scope with CONSEQUENCES: Causality, Counterfactuals, and Sequential Decision-Making for Recommender Systems. Traditionally, machine learning approaches to recommendation tend to cast it as a prediction task: “What is the probability that this user would like this product?”

Related content
In 2017, when the journal IEEE Internet Computing was celebrating its 20th anniversary, its editorial board decided to identify the single paper from its publication history that had best withstood the “test of time”. The honor went to a 2003 paper called “Amazon.com Recommendations: Item-to-Item Collaborative Filtering”, by then Amazon researchers Greg Linden, Brent Smith, and Jeremy York.

Recently, the community has begun to realize that this is a myopic way of looking at the problem, and that, rather than predictions, our systems should make decisions. This is an important distinction, as decisions have consequences, and the recommendations we provide can influence shoppers’ behavior, sellers’ exposure, future training data for our algorithms, and so on.

If we want to reason about the (possibly unintended) consequences of the decisions our machine learning models make, we need to borrow ideas from causal inference. Typical “what if”-type questions in learning and evaluation require a way to model counterfactuals. Accounting for causal factors can foster progress in effective, efficient and fair learning and evaluation from logged data.

CONSEQUENCES will host two invited speakers: a Nobel Prize–winning academic pioneer and an industrial research giant with over a decade of experience in both fundamental theory and applications of the topic at hand.

Guido Imbens

Guido cropped.png
Guido Imbens

Guido Imbens is the applied econometrics professor and professor of economics at the Stanford Graduate School of Business and a research consultant at Amazon. After graduating from Brown University, Guido taught at Harvard University, UCLA, and UC Berkeley. He joined Stanford in 2012. Imbens specializes in econometrics and, in particular, methods for drawing causal inferences. He is a fellow of the Econometric Society and the American Academy of Arts and Sciences. In 2021, he was co-awarded the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel for "methodological contributions to the analysis of causal relationships".

Lihong Li

Lihong cropped.png
Lihong Li

Lihong Li is a senior principal scientist at Amazon. He received a PhD in computer science from Rutgers University and has held research positions at Yahoo!, Microsoft, and Google. His main research interests are reinforcement learning, including contextual bandits, and related problems in AI. His work has found applications in recommendation, advertising, web search, and conversation systems, and he has won best-paper awards at ICML, AISTATS, and WSDM. He regularly serves as area chair or senior program committee member at major AI/ML conferences such as AAAI, AISTATS, ICLR, ICML, IJCAI, and NeurIPS.

The organizers of CONSEQUENCES+REVEAL are working together to implement a novel workshop format filled with exciting content, where the program will be divided over two days, based on topic. We will include a poster session for all accepted contributions, oral presentations for selected contributions, and an in-workshop tutorial at the start to introduce advanced concepts and techniques.
See the official workshop website for more information on how to submit your contributions. We look forward to seeing you there!

Important dates

Submission deadline

August 5, 2022

Author notification

August 27, 2022

Camera-ready-version deadline

September 10, 2022

CONSEQUENCES ’22

September 18–23, 2022

REVEAL ’22

September 18-23, 2022

Organizers

Related content
Amazon Scholar David Card and Amazon academic research consultant Guido Imbens talk about the past and future of empirical economics.

CONSEQUENCES

REVEAL

Related content

US, MA, North Reading
At Amazon Robotics, we design advanced robotic systems capable of intelligent perception, learning, and action alongside humans, all on a large scale. Our goal is to develop robots that increase productivity and efficiency at the Amazon fulfillment centers while ensuring the safety of workers. We are seeking an Applied Scientist to develop innovative, scalable solutions in feedback control and state estimation for robotic systems, with a focus on contact-rich manipulation tasks. In this role, you will formulate physics-based models of robotic systems, perform analytical and numerical studies, and design control and estimation algorithms that integrate fundamental principles with data-driven techniques. You will collaborate with a world-class team of experts in perception, machine learning, motion planning, and feedback controls to innovate and develop solutions for complex real-world problems. As part of your work, you will investigate applicable academic and industry research to develop, implement, and test solutions that support product features. You will also design and validate production designs. To succeed in this role, you should demonstrate a strong working knowledge of physical systems, a desire to learn from new challenges, and the problem-solving and communication skills to work within a highly interactive and experienced team. Candidates must show a hands-on passion for their work and the ability to communicate their ideas and concepts both verbally and visually. Key job responsibilities - Research, design, implement, and evaluate feedback control, estimation, and motion-planning algorithms, ensuring effective integration with perception, manipulation, and system-level components. - Develop experiments, simulations, and hardware prototypes to validate control algorithms, and optimization techniques in contact-rich manipulation and other challenging scenarios. - Collaborate with software engineering teams to enable scalable, real-time, and maintainable implementations of algorithms in production systems. - Partner with cross-functional teams across hardware, systems engineering, science, and operations to transition algorithms from early prototyping to robust, production-ready solutions. - Engage with stakeholders at all levels to iterate on system design, define requirements, and drive integration of control and estimation capabilities into Amazon Robotics platforms. A day in the life Amazon offers a full range of benefits that support you and eligible family members, including domestic partners and their children. Benefits can vary by location, the number of regularly scheduled hours you work, length of employment, and job status such as seasonal or temporary employment. The benefits that generally apply to regular, full-time employees include: 1. Medical, Dental, and Vision Coverage 2. Maternity and Parental Leave Options 3. Paid Time Off (PTO) 4. 401(k) Plan If you are not sure that every qualification on the list above describes you exactly, we'd still love to hear from you! At Amazon, we value people with unique backgrounds, experiences, and skillsets. If you’re passionate about this role and want to make an impact on a global scale, please apply!
IN, KA, Bengaluru
You will be working with a unique and gifted team developing exciting products for consumers. The team is a multidisciplinary group of engineers and scientists engaged in a fast paced mission to deliver new products. The team faces a challenging task of balancing cost, schedule, and performance requirements. You should be comfortable collaborating in a fast-paced and often uncertain environment, and contributing to innovative solutions, while demonstrating leadership, technical competence, and meticulousness. Your deliverables will include development of thermal solutions, concept design, feature development, product architecture and system validation through to manufacturing release. You will support creative developments through application of analysis and testing of complex electronic assemblies using advanced simulation and experimentation tools and techniques. Key job responsibilities In this role, you will: - Own thermal design for consumer electronics products at the system level, proposing thermal architecture and aligning with functional leads - Perform CFD simulations using tools such as Star-CCM+ or FloEFD to assess thermal feasibility, identify risks, and propose mitigation options - Generate data processing, statistical analysis, and test automation scripts to improve data consistency, insight quality, and team efficiency - Plan and execute thermal validation activities for devices and SoC packages, including test setup definition, data review, and issue tracking - Work closely with cross-functional and cross-geo teams to support product decisions, generate thermal specifications, and align on thermal requirements - Prepare clear summaries and reports on thermal results, risks, and observations for review by cross-functional leads About the team Amazon Lab126 is an inventive research and development company that designs and engineers high-profile consumer electronics. Lab126 began in 2004 as a subsidiary of Amazon.com, Inc., originally creating the best-selling Kindle family of products. Since then, we have produced innovative devices like Fire tablets, Fire TV and Amazon Echo. What will you help us create?
CA, ON, Toronto
The Sponsored Products and Brands team at Amazon Ads is re-imagining the advertising landscape through cutting-edge generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. Key job responsibilities • Collaborate with business, engineering and science leaders to establish science optimization and monetization roadmap for Amazon Retail Ad Service • Drive alignment across organizations for science, engineering and product strategy to achieve business goals • Lead/guide scientists and engineers across teams to develop, test, launch and improve of science models designed to optimize the shopper experience and deliver long term value for Amazon advertisers and third party retailers • Develop state of the art experimental approaches and ML models to keep up with our growing needs and diverse set of customers. • Participate in the Science hiring process as well as mentor other scientists - improving their skills, their knowledge of your solutions, and their ability to get things done. About the team Amazon Retail Ad Service within Sponsored Products and Brands is an ad-tech solution that enables retailers to monetize their online web and app traffic by displaying contextually relevant sponsored products ads. Our mission is to provide retailers with ad-solution for every type of supply to meet their advertising goals. At the same time, enable advertisers to manage their demand across multiple supplies (Amazon, offsite, third-party retailers) leveraging tools they are already familiar with. Our problem space is challenging and exciting in terms of different traffic patterns, varying product catalogs based on retailer industry and their shopper behaviors.
US, MA, N.reading
Amazon Industrial Robotics is seeking exceptional talent to help develop the next generation of advanced robotics systems that will transform automation at Amazon's scale. We're building revolutionary robotic systems that combine cutting-edge AI, sophisticated control systems, and advanced mechanical design to create adaptable automation solutions capable of working safely alongside humans in dynamic environments. This is a unique opportunity to shape the future of robotics and automation at an unprecedented scale, working with world-class teams pushing the boundaries of what's possible in robotic dexterous manipulation, locomotion, and human-robot interaction. This role presents an opportunity to shape the future of robotics through innovative applications of deep learning and large language models. At Amazon Industrial Robotics we leverage advanced robotics, machine learning, and artificial intelligence to solve complex operational challenges at an unprecedented scale. Our fleet of robots operates across hundreds of facilities worldwide, working in sophisticated coordination to fulfill our mission of customer excellence. We are pioneering the development of dexterous manipulation system that: - Enables unprecedented generalization across diverse tasks - Enables contact-rich manipulation in different environments - Seamlessly integrates low-level skills and high-level behaviors - Leverage mechanical intelligence, multi-modal sensor feedback and advanced control techniques. The ideal candidate will contribute to research that bridges the gap between theoretical advancement and practical implementation in robotics. You will be part of a team that's revolutionizing how robots learn, adapt, and interact with their environment. Join us in building the next generation of intelligent robotics systems that will transform the future of automation and human-robot collaboration. Key job responsibilities - Design and implement methods for dexterous manipulation - Design and implement methods for use of dexterous end effectors with force and tactile sensing - Develop a hierarchical system that combines low-level control with high-level planning - Utilize state-of-the-art manipulation models and optimal control techniques
AT, Graz
Are you a MS or PhD student interested in a 2026 internship in the field of machine learning, deep learning, generative AI, large language models and speech technology, robotics, computer vision, optimization, operations research, quantum computing, automated reasoning, or formal methods? If so, we want to hear from you! We are looking for students interested in using a variety of domain expertise to invent, design and implement state-of-the-art solutions for never-before-solved problems. You can find more information about the Amazon Science community as well as our interview process via the links below; https://www.amazon.science/ https://amazon.jobs/content/en/career-programs/university/science https://amazon.jobs/content/en/how-we-hire/university-roles/applied-science Key job responsibilities As an Applied Science Intern, you will own the design and development of end-to-end systems. You’ll have the opportunity to write technical white papers, create roadmaps and drive production level projects that will support Amazon Science. You will work closely with Amazon scientists and other science interns to develop solutions and deploy them into production. You will have the opportunity to design new algorithms, models, or other technical solutions whilst experiencing Amazon’s customer focused culture. The ideal intern must have the ability to work with diverse groups of people and cross-functional teams to solve complex business problems. A day in the life At Amazon, you will grow into the high impact person you know you’re ready to be. Every day will be filled with developing new skills and achieving personal growth. How often can you say that your work changes the world? At Amazon, you’ll say it often. Join us and define tomorrow. Some more benefits of an Amazon Science internship include; • All of our internships offer a competitive stipend/salary • Interns are paired with an experienced manager and mentor(s) • Interns receive invitations to different events such as intern program initiatives or site events • Interns can build their professional and personal network with other Amazon Scientists • Interns can potentially publish work at top tier conferences each year About the team Applicants will be reviewed on a rolling basis and are assigned to teams aligned with their research interests and experience prior to interviews. Start dates are available throughout the year and durations can vary in length from 3-6 months for full time internships. This role may available across multiple locations in the EMEA region (Austria, Estonia, France, Germany, Ireland, Israel, Italy, Jordan, Luxembourg, Netherlands, Poland, Romania, Spain, South Africa, UAE, and UK). Please note these are not remote internships.
IN, HR, Gurugram
Lead ML teams building large-scale forecasting and optimization systems that power Amazon’s global transportation network and directly impact customer experience and cost. As an Applied Science Manager, you will set scientific direction, mentor applied scientists, and partner with engineering and product leaders to deliver production-grade ML solutions at massive scale. Key job responsibilities 1. Lead and grow a high-performing team of Applied Scientists, providing technical guidance, mentorship, and career development. 2. Define and own the scientific vision and roadmap for ML solutions powering large-scale transportation planning and execution. 3. Guide model and system design across a range of techniques, including tree-based models, deep learning (LSTMs, transformers), LLMs, and reinforcement learning. 4. Ensure models are production-ready, scalable, and robust through close partnership with stakeholders. Partner with Product, Operations, and Engineering leaders to enable proactive decision-making and corrective actions. 5. Own end-to-end business metrics, directly influencing customer experience, cost optimization, and network reliability. 6. Help contribute to the broader ML community through publications, conference submissions, and internal knowledge sharing. A day in the life Your day includes reviewing model performance and business metrics, guiding technical design and experimentation, mentoring scientists, and driving roadmap execution. You’ll balance near-term delivery with long-term innovation while ensuring solutions are robust, interpretable, and scalable. Ultimately, your work helps improve delivery reliability, reduce costs, and enhance the customer experience at massive scale.
IL, Haifa
Come join the AWS Agentic AI science team in building the next generation models for intelligent automation. AWS, the world-leading provider of cloud services, has fostered the creation and growth of countless new businesses, and is a positive force for good. Our customers bring problems that will give Applied Scientists like you endless opportunities to see your research have a positive and immediate impact in the world. You will have the opportunity to partner with technology and business teams to solve real-world problems, have access to virtually endless data and computational resources, and to world-class engineers and developers that can help bring your ideas into the world. As part of the team, we expect that you will develop innovative solutions to hard problems, and publish your findings at peer reviewed conferences and workshops. We are looking for world class researchers with experience in one or more of the following areas - autonomous agents, API orchestration, Planning, large multimodal models (especially vision-language models), reinforcement learning (RL) and sequential decision making.
US, VA, Herndon
This position requires that the candidate selected be a US Citizen and currently possess and maintain an active Top Secret security clearance. The Amazon Web Services Professional Services (ProServe) team is seeking an experienced Delivery Practice Manager (DPM) to join our ProServe Shared Delivery Team (SDT) at Amazon Web Services (AWS). In this role, you'll manage a team of ProServe Delivery Consultants while supporting AWS enterprise customers through transformative projects. You'll leverage your IT and/or Management Consulting background to serve as a strategic advisor to customers, partners, and internal AWS teams. As a DPM you will be responsible for building and managing a team of Delivery Consultants and/or Engagement Managers working with customers and partners to architect and implement innovative solutions. You’ll routinely engage with Director, C-level executives, and governing boards, whilst being responsible for opportunity capture and driving engagement delivery. You’ll work closely with partner teams; drive business development initiatives through thought leadership; provide portfolio guidance and oversight; and meet and exceed customer satisfaction targets. As a DPM you are primarily focused directly or through their teams, on understanding and defining business outcomes for customers by building trust, identifying applicable AWS Professional Services offerings, and creating proposals and SOW’s. Your experience gained leading teams within the technology sector, will equip you with the ability to optimize team performance through implementing tailored people development plans, ensuring your teams are aligned to customer needs, and have the skills and capacity to address customer outcomes. Possessing the ability to translate technical concepts into business value for customers and then talk in technical depth with teams, you will cultivate strong customer, Amazon Global Sales (AGS), and ProServe team relationships which enables exceptional business performance. DPMs success is primarily measured by consistently delivering customer engagements by supporting sales through scoping technical requirements for an engagement, delivering engagements on time, within budget, and exceeding customer expectations. They will hold the Practice total utilization goal and be responsible for optimizing team performance. The AWS Professional Services organization is a global team of experts that help customers realize their desired business outcomes when using the AWS Cloud. We work together with customer teams and the AWS Partner Network (APN) to execute enterprise cloud computing initiatives. Our team provides assistance through a collection of offerings which help customers achieve specific outcomes related to enterprise cloud adoption. We also deliver focused guidance through our global specialty practices, which cover a variety of solutions, technologies, and industries. Key job responsibilities • Building and managing a high-performing team of Delivery Consultants • Collaborating with Delivery Consultants, Engagement Managers, Account Executives, and Cloud Architects to deploy solutions and provide input on new features • Developing and overseeing the implementation of innovative, forward-looking IT strategies for customers • Managing practice P&L, ensuring on-time and within-budget delivery of customer engagements • Driving business development initiatives and exceed customer satisfaction targets
IL, Haifa
Are you a scientist interested in pushing the state of the art in Information Retrieval, Large Language Models and Recommendation Systems? Are you interested in innovating on behalf of millions of customers, helping them accomplish their every day goals? Do you wish you had access to large datasets and tremendous computational resources? Do you want to join a team of capable scientist and engineers, building the future of e-commerce? Answer yes to any of these questions, and you will be a great fit for our team at Amazon. Our team is part of Amazon’s Personalization organization, a high-performing group that leverages Amazon’s expertise in machine learning, generative AI, large-scale data systems, and user experience design to deliver the best shopping experiences for our customers. Our team builds large-scale machine-learning solutions that delight customers with personalized and up-to-date recommendations that are related to their interests. We are a team uniquely placed within Amazon, to have a direct window of opportunity to influence how customers will think about their shopping journey in the future. As an Applied Scientist in our team, you will be responsible for the research, design, and development of new AI technologies for personalization. You will adopt or invent new machine learning and analytical techniques in the realm of recommendations, information retrieval and large language models. You will collaborate with scientists, engineers, and product partners locally and abroad. Your work will include inventing, experimenting with, and launching new features, products and systems. Please visit https://www.amazon.science for more information.
IL, Haifa
Are you a scientist interested in pushing the state of the art in Information Retrieval, Large Language Models and Recommendation Systems? Are you interested in innovating on behalf of millions of customers, helping them accomplish their every day goals? Do you wish you had access to large datasets and tremendous computational resources? Do you want to join a team of capable scientist and engineers, building the future of e-commerce? Answer yes to any of these questions, and you will be a great fit for our team at Amazon. Our team is part of Amazon’s Personalization organization, a high-performing group that leverages Amazon’s expertise in machine learning, generative AI, large-scale data systems, and user experience design to deliver the best shopping experiences for our customers. Our team builds large-scale machine-learning solutions that delight customers with personalized and up-to-date recommendations that are related to their interests. We are a team uniquely placed within Amazon, to have a direct window of opportunity to influence how customers will think about their shopping journey in the future. As an Applied Scientist in our team, you will be responsible for the research, design, and development of new AI technologies for personalization. You will adopt or invent new machine learning and analytical techniques in the realm of recommendations, information retrieval and large language models. You will collaborate with scientists, engineers, and product partners locally and abroad. Your work will include inventing, experimenting with, and launching new features, products and systems. Please visit https://www.amazon.science for more information.