The life of a prescription at Amazon Pharmacy

From pricing estimation and regulatory compliance to inventory management and chatbot assistants, machine learning models help Amazon Pharmacy customers stay healthy and save time and money.

Pharmacies play a vital role in ensuring patients’ health, but the process of dispensing medications is far more complex than it may appear. At Amazon Pharmacy, we are using artificial intelligence (AI) and cutting-edge technologies to remove this complexity and improve patients’ experiences.

The pharmacy challenge

When a prescription arrives at a pharmacy, its details must be entered into the pharmacy's software system. Then, a licensed pharmacist reviews the prescription to verify the patient's information, check for potential drug interactions or allergies, and confirm that the prescribed medication, dosage, and instructions are appropriate and accurate.

This process is susceptible to errors — even if the prescription arrives electronically. A U.S. study estimated that there are approximately 51.5 million dispensing errors annually in community pharmacies, with a meta-analysis supporting an error rate of around 1.5%.

Related content
Amazon Health Services' Sunita Mishra and Columbia University’s Katrina Armstrong discuss technology's potential role in medical settings.

Pharmacies must also handle billing and insurance claims for their patients. These are involved calculations based on patients’ specific insurance policies, their copay responsibilities, and the rates that the pharmacy has negotiated with different insurance providers. In fact, just identifying a patient's insurance can be challenging. The result is that patients often do not know the prices they will pay for their medications until the end of the process, when they’re picking up their prescriptions at a retail pharmacy or checking out online.

After a prescription has been validated and the purchase completed, the pharmacy staff must locate the specific medication in their inventory. However, it's possible that at this stage, the prescribed medication, the required strength, or the preferred brand may no longer be available. Providing a substitute could necessitate contacting the prescribing physician again for approval. Additionally, depending on the substitution, the billing and insurance process may need to be re-initiated to account for any changes in pricing or coverage.

Once the medication is ready for dispensing, the pharmacist provides the patient with detailed instructions on how to properly take it. Patients may also have questions regarding their insurance coverage or costs. However, these conversations often take place in public areas, which can be uncomfortable for patients who have personal or sensitive questions.

Finally, patients need convenient access to pharmacists at any time, day or night. This allows patients to report how they are feeling while taking their medications, which can help pharmacists provide better guidance and support throughout the treatment process.

The AI-powered pharmacy

Amazon Pharmacy uses large language models (LLMs) to enhance the accuracy, safety, and speed of prescription processing. First, we use LLMs to transcribe raw prescription data into structured, standardized formats that are seamlessly processed by software and more easily understood by patients. For example, medical abbreviations like "PRN" and "QID" are transformed into their full-text equivalents, such as "take as needed" and "take four times a day," respectively.

After standardizing the prescription data, the system performs a validation step that includes checking the medication names, dosage forms, strengths, and directions for use against an industry database. After validation, all prescriptions are still carefully reviewed and verified by licensed pharmacists. By leveraging this automated process, Amazon Pharmacy has reduced the number of near-miss events (potential medication errors) by 50% and improved processing speed by up to 90%. This allows our pharmacists to focus their time and attention on critical tasks, such as providing personalized care and addressing complex medication-related issues.

Pharmacy.png
The AI workflow at Amazon Pharmacy.

Amazon Pharmacy understands the importance of price transparency for customers. When a patient is using insurance to cover the cost of medication, Amazon Pharmacy will first try to obtain the exact price directly from the insurance provider. However, if this real-time pricing information is not available, Amazon Pharmacy will provide an estimated out-of-pocket cost for the patient's copay, without requiring the customer to go through the entire checkout process first.

To generate accurate price estimates, Amazon Pharmacy uses an ensemble of decision-tree-based models. These models take into account factors such as historical claims data (time series features) and static information such as the specific medication, the number-of-days' supply, and the quantity prescribed. By providing upfront pricing information, either the exact cost or a reliable estimate, Amazon Pharmacy aims to increase transparency and help patients understand their out-of-pocket expenses before committing to purchases. Additionally, Amazon Pharmacy searches for applicable industry coupons and automatically applies them to orders. We also use ML to validate the patient's insurance registration and claim requests to insurance providers.

Amazon is known for its extensive logistics and fulfillment capabilities, and Amazon Pharmacy takes advantage of Amazon's vast network of same-day and local delivery facilities, as well as innovative transportation methods like Prime Air drones. Additionally, Amazon Pharmacy employs specialized automation technologies, such as robotic vial-filling systems, to streamline the medication-dispensing process, enabling prompt delivery of medications to patients across the nation.

Beyond the physical logistics infrastructure, Amazon Pharmacy has developed its own order fulfillment system to handle complex medication-routing and -dispensing logic, while ensuring compliance with over 160 different pharmacy regulatory bodies across the United States. For example, if a medication for an order is no longer available at the closest fulfillment center, Amazon Pharmacy can identify the next best eligible facility to fulfill the order, even if it's in a different state, provided that the relevant state regulations allow for such cross-state fulfillment.

Related content
ARA recipient Marinka Zitnik is focused on how machine learning can enable accurate diagnoses and the development of new treatments and therapies.

Moreover, Amazon Pharmacy's order fulfillment algorithm takes into account regional variations in insurance eligibility. In such cases, Amazon Pharmacy will first validate that there are no changes to the patient's copay. If changes are required, Amazon Pharmacy will work with the insurance providers to clarify the benefits. To accomplish all of this, Amazon Pharmacy's order fulfillment solution employs a combination of operations research techniques, such as optimization solvers, and deep-learning models such as variational autoencoders and diffusion models. These models help simulate different scenarios and optimize the fulfillment process to ensure efficient and compliant delivery of medications to patients.

Amazon Pharmacy also introduced personalized AI-powered chatbots to assist users. These virtual assistants can answer frequently asked questions about Amazon Pharmacy, such as how to enroll in the service. In a first for the industry, Amazon Pharmacy's chatbot also provides personalized support, allowing patients to ask questions about their medication orders, delivery status, prescription transfers, and inventory availability. If a patient prefers, there is always 24/7 access to direct pharmacist support and the customer care team.

Implementing a personalized AI chatbot in the healthcare setting is a complex task. It's crucial to safeguard patients' privacy and ensure the highest level of accuracy, avoiding LLM hallucinations. To address these challenges, Amazon Pharmacy has enhanced the typical retrieval-augmented generation (RAG) approach used for LLM chatbots. The enhancements include input and output guardrails, the use of ensembles of specialized (mini) AI models, and a continuous model improvement process through reinforcement learning using human feedback (RLHF).

The digital pharmacy counter

Amazon Pharmacy is leveraging ML and optimization algorithms to streamline the complex process of dispensing medications. By addressing long-standing challenges such as data entry errors, lack of price transparency, intelligent nationwide medication fulfillment, and personalized-AI-based experiences, Amazon Pharmacy enables patients to save time, save money, and stay healthy.

Related content
Gari Clifford, the chair of the Department of Biomedical Informatics at Emory University and an Amazon Research Award recipient, wants to transform healthcare.

For example, it is not uncommon for medications to go into short-term back order (STBO), especially new medications entering the market, as was recently the case with the GLP-1 line of medications used for diabetes and weight loss. Amazon Pharmacy’s intelligent-fulfillment solution has enabled an 85% decrease in delivery estimate misses for unforeseen reasons, including STBOs.

The AI-powered Amazon Pharmacy assistant helps customers navigate the complexities of the pharmacy industry, providing 24/7 assistance on topics like prescription tracking, insurance coverage, medication availability, and cost-saving strategies. Half of the customers who interact with the assistant don't require additional human support, which saves them time and effort. (For customers who still need assistance, Amazon Pharmacy likewise provides 24/7 access to pharmacist support.) Additionally, the assistant provides real-time medication transfer or shipment status updates in response to patient queries, handling follow-up questions to recommend next steps.

For all the success of our AI-based systems, however, Amazon Pharmacy’s research and engineering teams remain hard at work. We will continue pushing the envelope in scaling medication dispensing, improving personalized AI-based chatbots and assistants, and transitioning toward a longitudinal pharmacy that is proactively looking out for patients.

Research areas

Related content

ES, B, Barcelona
Are you interested in defining the science strategy that enables Amazon to market to millions of customers based on their lifecycle needs rather than one-size-fits-all campaigns? We are seeking a Applied Scientist to lead the science strategy for our Lifecycle Marketing Experimentation roadmap within the PRIMAS (Prime & Marketing analytics and science) team. The position is open to candidates in Amsterdam and Barcelona. In this role, you will own the end-to-end science approach that enables EU marketing to shift from broad, generic campaigns to targeted, cohort-based marketing that changes customer behavior. This is a high-ambiguity, high-impact role where you will define what problems are worth solving, build the science foundation from scratch, and influence senior business leaders on marketing strategy. You will work directly with Business Directors and channel leaders to solve critical business problems: how do we win back customers lost to competitors, convert Young Adults to Prime, and optimize marketing spend by de-averaging across customer cohorts. Key job responsibilities Science Strategy & Leadership: 1. Own the end-to-end science strategy for lifecycle marketing, defining the roadmap across audience targeting, behavioral modeling, and measurement 2. Navigate high ambiguity in defining customer journey frameworks and behavioral models – our most challenging science problem with no established playbook 3. Lead strategic discussions with business leaders translating business needs into science solutions and building trust across business and tech partners 4. Mentor and guide a team of 2-3 scientists and BIEs on technical execution while contributing hands-on to the hardest problems Advanced Customer Behavior Modeling: 1. Build sophisticated propensity models identifying customer cohorts based on lifecycle stage and complex behavioral patterns (e.g., Bargain hunters, Young adults Prime prospects) 2. Define customer journey frameworks using advanced techniques (Hidden Markov Models, sequential decision-making) to model how customers transition across lifecycle stages 3. Identify which customer behaviors and triggers drive lifecycle progression and what messaging/levers are most effective for each cohort 4. Integrate 1P behavioral data with 2P survey insights to create rich, actionable audience definitions Measurement & Cross-Workstream Integration: 1. Partner with measurement scientist to design experiments (RCTs) that isolate audience targeting effects from creative effects 2. Ensure audience definitions, journey models, and measurement frameworks work coherently across Meta, LiveRamp, and owned channels 3. Establish feedback loops connecting measurement insights back to model improvements About the team The PRIMAS (Prime & Marketing Analytics and Science) is the team that support the science & analytics needs of the EU Prime and Marketing organization, an org that supports the Prime and Marketing programs in European marketplaces and comprises 250-300 employees. The PRIMAS team, is part of a larger tech tech team of 100+ people called WIMSI (WW Integrated Marketing Systems and Intelligence). WIMSI core mission is to accelerate marketing technology capabilities that enable de-averaged customer experiences across the marketing funnel: awareness, consideration, and conversion.
IN, KA, Bengaluru
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced algorithmic systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning and Data Sciences team for India Consumer Businesses. If you have an entrepreneurial spirit, know how to deliver, love to work with data, are deeply technical, highly innovative and long for the opportunity to build solutions to challenging problems that directly impact the company's bottom-line, we want to talk to you. Major responsibilities - Use machine learning and analytical techniques to create scalable solutions for business problems - Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes - Design, development, evaluate and deploy innovative and highly scalable models for predictive learning - Research and implement novel machine learning and statistical approaches - Work closely with software engineering teams to drive real-time model implementations and new feature creations - Work closely with business owners and operations staff to optimize various business operations - Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model implementation - Mentor other scientists and engineers in the use of ML techniques
ES, M, Madrid
At Amazon, we are committed to being the Earth's most customer-centric company. The European International Technology group (EU INTech) owns the enhancement and delivery of Amazon's engineering to all the varied customers and cultures of the world. We do this through a combination of partnerships with other Amazon technical teams and our own innovative new projects. You will be joining the Tamale team to work on Haul. As part of EU INTech and Haul, Tamale strives to create a discovery-driven shopping experience using challenging machine learning and ranking solutions. You will be exposed to large-scale recommendation systems, multi-objective optimization, and state-of-the-art deep learning architectures, and you'll be part of a key effort to improve our customers' browsing experience by building next-generation ranking models for Amazon Haul's endless scroll experience. We are looking for a passionate, talented, and inventive Scientist with a strong machine learning background to help build industry-leading ranking solutions. We strongly value your hard work and obsession to solve complex problems on behalf of Amazon customers. Key job responsibilities We look for applied scientists who possess a wide variety of skills. As the successful applicant for this role, you will work closely with your business partners to identify opportunities for innovation. You will apply machine learning solutions to optimize multi-objective ranking, improve discovery engagement through contextual signals, and scale ranking systems across multiple marketplaces. You will work with business leaders, scientists, and product managers to translate business and functional requirements into concrete deliverables, including the design, development, testing, and deployment of highly scalable distributed ranking services. You will be part of a team of scientists and engineers working on solving ranking and personalization challenges at scale. You will be able to influence the scientific roadmap of the team, setting the standards for scientific excellence. You will be working with state-of-the-art architectures and real-time feature serving systems. Your work will improve the experience of millions of daily customers using Amazon Haul worldwide. You will have the chance to have great customer impact and continue growing in one of the most innovative companies in the world. You will learn a huge amount - and have a lot of fun - in the process!
IN, HR, Gurugram
Do you want to join an innovative team of scientists who use machine learning and statistical techniques to create state-of-the-art solutions for providing better value to Amazon’s customers? Do you want to build and deploy advanced ML systems that help optimize millions of transactions every day? Are you excited by the prospect of analyzing and modeling terabytes of data to solve real-world problems? Do you like to own end-to-end business problems/metrics and directly impact the profitability of the company? Do you like to innovate and simplify? If yes, then you may be a great fit to join the Machine Learning team for International Emerging Stores (IES). Machine Learning, Big Data and related quantitative sciences have been strategic to Amazon from the early years. Amazon has been a pioneer in areas such as recommendation engines, ecommerce fraud detection and large-scale optimization of fulfillment center operations. As Amazon has rapidly grown and diversified, the opportunity for applying machine learning has exploded. We have a very broad collection of practical problems where machine learning systems can dramatically improve the customer experience, reduce cost, and drive speed and automation. These include product bundle recommendations for millions of products, safeguarding financial transactions across by building the risk models, improving catalog quality via extracting product attribute values from structured/unstructured data for millions of products, enhancing address quality by powering customer suggestions We are developing state-of-the-art machine learning solutions to accelerate the Amazon India growth story. Amazon is an exciting place to be at for a machine learning practitioner. We have the eagerness of a fresh startup to absorb machine learning solutions, and the scale of a mature firm to help support their development at the same time. As part of the International Machine Learning team, you will get to work alongside brilliant minds motivated to solve real-world machine learning problems that make a difference to millions of our customers. We encourage thought leadership and blue ocean thinking in ML. Key job responsibilities Use machine learning and analytical techniques to create scalable solutions for business problems Analyze and extract relevant information from large amounts of Amazon’s historical business data to help automate and optimize key processes Design, develop, evaluate and deploy, innovative and highly scalable ML models Work closely with software engineering teams to drive real-time model implementations Work closely with business partners to identify problems and propose machine learning solutions Establish scalable, efficient, automated processes for large scale data analyses, model development, model validation and model maintenance Work proactively with engineering teams and product managers to evangelize new algorithms and drive the implementation of large-scale complex ML models in production Leading projects and mentoring other scientists, engineers in the use of ML techniques About the team International Machine Learning Team is responsible for building novel ML solutions across International Emerging Store (India, MENA, Far-East, LatAm) problems and impact the bottom-line and top-line of India business. Learn more about our team from https://www.amazon.science/working-at-amazon/how-rajeev-rastogis-machine-learning-team-in-india-develops-innovations-for-customers-worldwide
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, WA, Bellevue
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
US, MA, Boston
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems to ensure the highest standards of data quality, to build industry-leading technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As part of the AGI team, an Applied Scientist will collaborate closely with core scientist team developing Amazon Nova models. They will lead the development of comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. This includes designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that help Nova improve performances on benchmarks. The Applied Scientist will perform expert-level manual audits, conduct meta-audits to evaluate auditor performance, and provide targeted coaching to uplift overall quality capabilities. A critical aspect of this role involves developing and maintaining LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Scientist will also set up the configuration of data collection workflows and communicate quality feedback to stakeholders. An Applied Scientist will also have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services. A day in the life An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.