Preskill wins prize for work on learning and quantum computing

Caltech professor and Amazon Scholar John Preskill wins Bell Prize for applying both classical and quantum computing to the problem of learning from quantum experiments.

John Preskill
John Preskill, the Richard P. Feynman Professor of Theoretical Physics at the California Institute of Technology and an Amazon Scholar.
Credit: Caltech / Lance Hayashida

In August 2024, at the 10th International Conference on Quantum Information and Quantum Control, John Preskill, the Richard P. Feynman Professor of Theoretical Physics at the California Institute of Technology and an Amazon Scholar, will receive the John Stewart Bell Prize for Research on Fundamental Issues in Quantum Mechanics and Their Applications. The prize is named for the European Organization for Nuclear Research (CERN) physicist John Bell, who showed how to prove the existence of quantum entanglement, or strong correlations between the physical states of quantum systems, even when they’re separated by great distances.

The prize announcement cites Preskill’s work “at the interface of efficient learning and processing of quantum information in quantum computation”, work that explores both classical and quantum techniques for using machine learning to deepen our understanding of quantum systems. Preskill recently took some time to explain his prize-winning work to Amazon Science.

  1. Q. 

    Can you describe the work that won the prize?

    A. 

    You could put it in two categories, which we could call learning about the quantum world using classical machines and using quantum machines. People have quantum computers now with hundreds of quantum bits, or qubits, and completely characterizing the state of a quantum computer with hundreds of qubits is beyond our ability, because that complete description grows exponentially with the number of qubits.

    If we're going to make progress, we have to have some way of translating that quantum information to classical information that we can understand. So part of our work — and this was with two brilliant collaborators, Robert Huang, a student, and Richard Kueng, a postdoc — was a way of translating this very complex quantum system to a succinct classical description.

    What we showed is that there's a way of doing a relatively modest number of experiments that gives you a description of the quantum system from which you can predict very many properties — a lot more properties than the number of measurements that you had to make. We call this description a “classical shadow”.

    An irregular polyhedron suspended in midair, with shadows projected onto each of three orthogonal surfaces: one shadow is a triangle, one a square, and the third a circle.
    Computing "classical shadows" is analogous to projecting a 3-D object into two dimensions along multiple axes.

    Let's say there's a three-dimensional object, and we're trying to understand its geometry. We can take snapshots of it from different directions, which project it on two dimensions. This is kind of like that only on steroids, because the quantum system lives in some unimaginably large dimension, and we're projecting it down to a little bit of information. What we showed is that you don't need so many of these snapshots to be able to predict a lot of things that a physicist would typically be interested in.

    We'd like to use the data that we get from quantum experiments and generalize to predict what we'll see when we look at related quantum systems or when we look at the same quantum system in a different way. And you know, AI is everywhere these days, and a lot of people are thinking about applying machine learning to understanding quantum systems. But it's mostly very heuristic: people try different things, and they hope that gives them the ability to generalize and make good predictions.

    From left to right, three phases in the process of learning from quantum systems: at left is a depiction of atoms in a sphere, labeled "unknown quantum system"; in the middle is a rendering of binary values on computer screen, labeled "efficient classical representation"; and at right is the same computer screen, displaying different-colored probability distributions, labeled "predict properties".
    The computational pipeline for learning about quantum systems with classical computers.

    What we wanted to do is to give rigorous performance guarantees that you don't need that many of these snapshots in order to generalize with a small error. And we were able to prove that in some settings.

    When it comes to learning with quantum machines, now let's do something different. Let's grab some quantum data — maybe we produce it on a quantum computer, or we have a sensing network that collected some photons from somewhere — and store that in a quantum memory. We don’t just measure it and put it in a classical memory; we store it in a quantum memory, and then we do a quantum computation on that data. And finally, at the end of the computation, we get a classical answer, because at the end of a quantum computation, you always do.

    What we were able to show is that, for some properties of the quantum system that you might want to know, it's vastly more efficient to process with a quantum computer than a classical computer.

  2. Q. 

    In the case of the “classical shadows”, do you have to reset the system after each measurement?

    A. 

    I'm imagining a scenario in which I have access to many identically prepared copies. Now, I might have prepared them with a quantum computer, and I went through the same steps of the computation each time. Or maybe there was some experiment I did in the lab, which I can repeat over and over again. The main point of our work was, you don't need as many copies as you thought you might. Technically, the number of predictions we can make with some fixed accuracy, based on measuring the same state many times, is exponential in the number of copies that we measure.

  3. Q. 

    Do you have to know what questions you want to ask before you start making measurements?

    A. 

    We have a slogan, which is “Measure first, ask questions later”, because it turns out that no, you don't need to know what properties you're going to want to learn at the time that you make the measurements. And as a result, the measurements that we require for creating a classical shadow really are experimentally feasible today, because all you have to do is measure the individual qubits.

    Related content
    The noted physicist answers 3 questions about the challenges of quantum computing and why he’s excited to be part of a technology development project.

    The trick is you measure them in a random basis. There are different ways of looking at a qubit. You can, so to speak, look at it straight up and down or horizontally or back and forth. So there are three types of measurements we consider, and for all the qubits, we randomly choose to measure in one of those three ways.

    There's some power that comes from the randomness there. Later, you can say, Okay, I want to use that data and predict something like a correlation function for a clump of qubits here and a clump there, or maybe the expectation value of the energy of some quantum system, and just by processing that randomized data, I can make that prediction.

  4. Q. 

    What’s the setup in the quantum learning context?

    A. 

    The quantum setting is you can take two copies at once, store them in a quantum memory, and then do a computation across the two copies. We call that an entangled or entangling measurement of the two copies. And that's where the power comes from. When you do an entangling measurement on two copies at a time, that enables us to, in some cases, vastly reduce the number of experiments we need to do to predict the properties.

    Of course, in a real computer, there's noise, which is always a factor. But if everything's noiseless, then for the particular case that we studied, the number of measurements that suffice when you do these entangling measurements across the copies is a constant. It doesn't depend on how large the system is. But if you measure one copy at a time, the theorem says that to get that same measurement accuracy, you'd have to measure a number of copies which is exponential in the number of qubits.

  5. Q. 

    What applications could this have?

    A. 

    What we imagine doing eventually, which I think will be very empowering, is a new kind of quantum sensing. If we are observing light from some source, what do we do now? We count photons, typically: with a camera, you've got pixels that flash when they get hit by photons.

    Related content
    Research on “super-Grover” optimization, quantum algorithms for topological data analysis, and simulation of physical systems displays the range of Amazon’s interests in quantum computing.

    If there's a state of many photons that has come from some source — maybe you’ve got telescopes, and you're looking at something coming in from space — there's a lot of information, at least in principle, in the quantum correlations among those photons. And we miss that if we're just counting photons. You're throwing away a tremendous amount of information in that many-photon state.

    What we can imagine, when we have the technology to do it, is our telescopes won't just count photons. They'll collect this many-photon state and store it in a quantum memory, including multiple images, and then we can come along and do this collective measurement on the multiple copies. And we'll be able to see things in that signal that we would just miss if we do things the conventional way, measuring one copy at a time.

  6. Q. 

    One last question: the prize is named for John Bell, who proposed an experiment to prove that measurements on entangled particles really do depend on each other, even if the particles are separated by enormous distances. Does your work relate to Bell’s in some way?

    A. 

    The charge to the committee that selects the awardee is to identify research that advances the foundations of quantum theory. And of course, Bell did that by formulating Bell inequalities showing that quantum entanglement enables us to do things that we couldn't do without quantum entanglement. That's the point of experimental demonstrations of violations of Bell's inequality.

    Part of Bell's legacy is that quantum entanglement is a resource that, if we know how to make use of it, enables us to do things we couldn't otherwise do — more-powerful computations, new kinds of measurements, new kinds of communication. So I think, at least in that sense, the work we've been talking about is very much following in Bell's footsteps — as is the whole field of quantum computing, in a way. Because I think the power of quantum computers really comes from the feature that in the middle of a quantum computation, you're dealing with a very highly entangled state of many qubits that we don't know how to represent classically.

Research areas

Related content

ES, B, Barcelona
Are you interested in defining the science strategy that enables Amazon to market to millions of customers based on their lifecycle needs rather than one-size-fits-all campaigns? We are seeking a Applied Scientist to lead the science strategy for our Lifecycle Marketing Experimentation roadmap within the PRIMAS (Prime & Marketing analytics and science) team. The position is open to candidates in Amsterdam and Barcelona. In this role, you will own the end-to-end science approach that enables EU marketing to shift from broad, generic campaigns to targeted, cohort-based marketing that changes customer behavior. This is a high-ambiguity, high-impact role where you will define what problems are worth solving, build the science foundation from scratch, and influence senior business leaders on marketing strategy. You will work directly with Business Directors and channel leaders to solve critical business problems: how do we win back customers lost to competitors, convert Young Adults to Prime, and optimize marketing spend by de-averaging across customer cohorts. Key job responsibilities Science Strategy & Leadership: 1. Own the end-to-end science strategy for lifecycle marketing, defining the roadmap across audience targeting, behavioral modeling, and measurement 2. Navigate high ambiguity in defining customer journey frameworks and behavioral models – our most challenging science problem with no established playbook 3. Lead strategic discussions with business leaders translating business needs into science solutions and building trust across business and tech partners 4. Mentor and guide a team of 2-3 scientists and BIEs on technical execution while contributing hands-on to the hardest problems Advanced Customer Behavior Modeling: 1. Build sophisticated propensity models identifying customer cohorts based on lifecycle stage and complex behavioral patterns (e.g., Bargain hunters, Young adults Prime prospects) 2. Define customer journey frameworks using advanced techniques (Hidden Markov Models, sequential decision-making) to model how customers transition across lifecycle stages 3. Identify which customer behaviors and triggers drive lifecycle progression and what messaging/levers are most effective for each cohort 4. Integrate 1P behavioral data with 2P survey insights to create rich, actionable audience definitions Measurement & Cross-Workstream Integration: 1. Partner with measurement scientist to design experiments (RCTs) that isolate audience targeting effects from creative effects 2. Ensure audience definitions, journey models, and measurement frameworks work coherently across Meta, LiveRamp, and owned channels 3. Establish feedback loops connecting measurement insights back to model improvements About the team The PRIMAS (Prime & Marketing Analytics and Science) is the team that support the science & analytics needs of the EU Prime and Marketing organization, an org that supports the Prime and Marketing programs in European marketplaces and comprises 250-300 employees. The PRIMAS team, is part of a larger tech tech team of 100+ people called WIMSI (WW Integrated Marketing Systems and Intelligence). WIMSI core mission is to accelerate marketing technology capabilities that enable de-averaged customer experiences across the marketing funnel: awareness, consideration, and conversion.
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses. If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you. Major responsibilities - Use machine learning and analytical techniques to create scalable solutions for business problems - Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes - Design, development, evaluate and deploy innovative and highly scalable models for predictive learning - Research and implement novel machine learning and statistical approaches - Work closely with software engineering teams to drive real-time model implementations and new feature creations - Work closely with business owners and operations staff to optimize various business operations - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Mentor other scientists and engineers in the use of ML techniques
ES, M, Madrid
At Amazon, we are committed to being the Earth's most customer-centric company. The European International Technology group (EU INTech) owns the enhancement and delivery of Amazon's engineering to all the varied customers and cultures of the world. We do this through a combination of partnerships with other Amazon technical teams and our own innovative new projects. You will be joining the Tamale team to work on Haul. As part of EU INTech and Haul, Tamale strives to create a discovery-driven shopping experience using challenging machine learning and ranking solutions. You will be exposed to large-scale recommendation systems, multi-objective optimization, and state-of-the-art deep learning architectures, and you'll be part of a key effort to improve our customers' browsing experience by building next-generation ranking models for Amazon Haul's endless scroll experience. We are looking for a passionate, talented, and inventive Scientist with a strong machine learning background to help build industry-leading ranking solutions. We strongly value your hard work and obsession to solve complex problems on behalf of Amazon customers. Key job responsibilities We look for applied scientists who possess a wide variety of skills. As the successful applicant for this role, you will work closely with your business partners to identify opportunities for innovation. You will apply machine learning solutions to optimize multi-objective ranking, improve discovery engagement through contextual signals, and scale ranking systems across multiple marketplaces. You will work with business leaders, scientists, and product managers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed ranking services. You will be part of a team of scientists and engineers working on solving ranking and personalization challenges at scale. You will be able to influence the scientific roadmap of the team, setting the standards for scientific excellence. You will be working with state-of-the-art architectures and real-time feature serving systems. Your work will improve the experience of millions of daily customers using Amazon Haul worldwide. You will have the chance to have great customer impact and continue growing in one of the most innovative companies in the world. You will learn a huge amount - and have a lot of fun - in the process!
IN, HR, Gurugram
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced ML systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real-world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning team for International Emerging Stores (IES). Machine Learning, Big Data and related quantitative sciences have been strategic to Amazon from the early years. Amazon has been a pioneer in areas such as recommendation engines, ecommerce fraud detection and large-scale optimization of fulfillment center operations. As Amazon has rapidly grown and diversified, the opportunity for applying machine learning has exploded. We have a very broad collection of practical problems where machine learning systems can dramatically improve the customer experience, reduce cost, and drive speed and automation. These include product bundle recommendations for millions of products, safeguarding financial transactions across by building the risk models, improving catalog quality via extracting product attribute values from structured/unstructured data for millions of products, enhancing address quality by powering customer suggestions We are developing state-of-the-art machine learning solutions to accelerate the Amazon India growth story. Amazon is an exciting place to be at for a machine learning practitioner. We have the eagerness of a fresh startup to absorb machine learning solutions, and the scale of a mature firm to help support their development at the same time. As part of the International Machine Learning team, you will get to work alongside brilliant minds motivated to solve real-world machine learning problems that make a difference to millions of our customers. We encourage thought leadership and blue ocean thinking in ML. Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes Design, develop, evaluate and deploy, innovative and highly scalable ML models Work closely with software engineering teams to drive real-time model implementations Work closely with business partners to identify problems and propose machine learning solutions Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model maintenance Work proactively with engineering teams and product managers to evangelize new algorithms and drive the implementation of large-scale complex ML models in production Leading projects and mentoring other scientists, engineers in the use of ML techniques About the team International Machine Learning Team is responsible for building novel ML solutions across International Emerging Store (India, MENA, Far-East, LatAm) problems and impact the bottom-line and top-line of India business. Learn more about our team from https://www.amazon.science/working-at-amazon/how-rajeev-rastogis-machine-learning-team-in-india-develops-innovations-for-customers-worldwide
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, WA, Bellevue
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.