Social
Publication

DiffSign: AI-assisted generation of customizable sign language videos with enhanced realism

By Sudha Krishnamurthy, Vimal Bhat, Abhinav Jain
2024
Download Copy BibTeX
Share
Download
Copy BibTeX
Share
The proliferation of several streaming services in recent years has now made it possible for a diverse audience across the world to view the same media content, such as movies or TV shows. While translation and dubbing services are being added to make content accessible to the local audience, the support for making content accessible to people with different abilities, such as the Deaf and Hard of Hearing (DHH) community, is still lagging. Our goal is to make media content more accessible to the DHH community by generating sign language videos with synthetic signers that are realistic and expressive. Using the same signer for a given media content that is viewed globally may have limited appeal. Hence, our approach combines parametric modeling and generative modeling to generate realistic-looking synthetic signers and customize their appearance based on user preferences. We first retarget human sign language poses to 3D sign language avatars by optimizing a parametric model. The high-fidelity poses from the rendered avatars are then used to condition the poses of synthetic signers generated using a diffusion-based generative model. The appearance of the synthetic signer is controlled by an image prompt supplied through a visual adapter. Our results show that the sign language videos generated using our approach have better temporal consistency and realism than signing videos generated by a diffusion model conditioned only on text prompts. We also support multimodal prompts to allow users to further customize the signer’s appearance to accommodate diversity (e.g. skin tone, gender). Our approach is also useful for signer anonymization.
Research areas
Tags

Latest news

Work with us

See more jobs See more jobs
Member of Technical Staff, Artificial General Intelligence
US, CA, Sunnyvale
The Artificial General Intelligence (AGI) team is looking for a passionate, talented, and inventive Member of Technical Staff with a strong deep learning background, to build industry-leading Generative Artificial Intelligence (GenAI) technology with Large Language Models (LLMs) and multimodal systems. Key job responsibilities As a Member of Technical Staff with the AGI team, you will lead the development of algorithms and modeling techniques, to advance the state of the art with LLMs. You will lead the foundational model development in an applied research role, including model training, dataset design, and pre- and post-training optimization. Your work will directly impact our customers in the form of products and services that make use of GenAI technology. You will leverage Amazon’s heterogeneous data sources and large-scale computing resources to accelerate advances in LLMs. About the team The AGI team has a mission to push the envelope in GenAI with LLMs and multimodal systems, in order to provide the best-possible experience for our customers.
Economist, Ads Econ
US, NY, New York
Amazon Advertising is one of Amazon's fastest growing and most profitable businesses. Our products are used daily to surface new selection and provide customers a wider set of product choices along their shopping journeys. The business is focused on generating value for shoppers as well as advertisers. Our team uses a combination of econometrics, machine learning, and data science to build disruptive products for all our Advertising products. We also generate insights to guide Amazon Advertising strategy, providing direct support to senior leadership. We are looking for an experienced Economist with a deep passion for building econometric solutions and the ability to communicate data insights and scientific vision to execute on strategic projects. Key job responsibilities - Leverage econometrics and ML models to optimize advertising strategies on behalf of our customers. - Influence key business and product decisions based on insights from models you develop. - Perform hands-on analysis and modeling with enormous data sets to develop insights that increase traffic monetization and merchandise sales without compromising shopper experience. - Work closely with software engineers on detailed requirements to productionize the models you build. - Run A/B experiments that affect hundreds of millions of customers, evaluate the impact of your optimizations and communicate your results to various business stakeholders. - Work with other scientists, software developers, and product partners to implement your solutions.
Economist , Economic Decision Science
GB, London
We are looking for an Economist to work on exciting and challenging business problems related to Amazon Retail’s worldwide product assortment. You will build innovative solutions based on econometrics, machine learning, and experimentation. You will be part of a interdisciplinary team of economists, product managers, engineers, and scientists, and your work will influence finance and business decisions affecting Amazon’s vast product assortment globally. If you have an entrepreneurial spirit, you know how to deliver results fast, and you have a deeply quantitative, highly innovative approach to solving problems, and long for the opportunity to build pioneering solutions to challenging problems, we want to talk to you. Key job responsibilities * Work on a challenging problem that has the potential to significantly impact Amazon’s business position * Develop econometric models and experiments to measure the customer and financial impact of Amazon’s product assortment * Collaborate with other scientists at Amazon to deliver measurable progress and change * Influence business leaders based on empirical findings
Senior Applied Scientist, Sponsored Products and Brands
US, NY, New York
About Sponsored Products and Brands: The Sponsored Products and Brands (SPB) organization at Amazon Ads is re-imagining the advertising landscape through industry leading generative AI technologies, revolutionizing how millions of customers discover products and engage with brands across Amazon.com and beyond. We are at the forefront of re-inventing advertising experiences, bridging human creativity with artificial intelligence to transform every aspect of the advertising lifecycle from ad creation and optimization to performance analysis and customer insights. We are a passionate group of innovators dedicated to developing responsible and intelligent AI technologies that balance the needs of advertisers, enhance the shopping experience, and strengthen the marketplace. If you're energized by solving complex challenges and pushing the boundaries of what's possible with AI, join us in shaping the future of advertising. About Our Team: The Brand Beacon team is responsible for inventing impressions offerings for brands to increase share of voice via premium experiences, helping brands get discovered, acquire new customers and sustainably grow customer lifetime value. We build end-to-end solutions that enable brands to drive discovery, visibility and share of voice. This includes building advertiser controls, shopper experiences, monetization strategies and optimization features. We succeed when (1) shoppers discover, engage and build affinity with brands and (2) brands can grow their business at scale with our advertising products. About This Role: As a Senior Scientist for the team, you will have the opportunity to apply your deep subject matter expertise in the area of ML, LLM and GenAI models. You will invent new product experiences that enable novel advertiser and shopper experiences. This role will liaise with internal Amazon partners and work on bringing state-of-the-art GenAI models to production, and stay abreast of the latest developments in the space of GenAI and identify opportunities to improve the efficiency and productivity of the team. Additionally, you will define a long-term science vision for our advertising business, driven by our customer’s needs, and translate it into actionable plans for our team of applied scientists and engineers. This role will play a critical role in elevating the team’s scientific and technical rigor, identifying and implementing best-in-class algorithms, methodologies, and infrastructure that enable rapid experimentation and scaling. You will communicate learnings to leadership and mentor and grow Applied AI talent across org. * Develop AI solutions for advertiser and shopper experiences. Build monetization and optimization systems that leverage generative models to value and improve campaign performance. * Define a long-term science vision and roadmap for our advertising business, driven from our customers' needs, translating that direction into specific plans for applied scientists and engineering teams. This role combines science leadership, organizational ability, technical strength, product focus, and business understanding. * Design and conduct A/B experiments to evaluate proposed solutions based on in-depth data analyses. * Effectively communicate technical and non-technical ideas with teammates and stakeholders. * Stay up-to-date with advancements and the latest modeling techniques in the field. * Think big about the arc of development of Gen AI over a multi-year horizon and identify new opportunities to apply these technologies to solve real-world problems. #GenAI
Applied Science Manager, Stores-Ads Science
US, WA, Seattle
Amazon's Stores-Ads Science team operates at the intersection of Amazon's Stores and advertising businesses. We develop causal measurement systems, optimization algorithms, and machine learning models that inform how advertising affects shopper engagement, driving selling partner growth and marketplace economics. Our science shapes decisions both at the strategic level and in production systems. We are a team of interdisciplinary scientists who combine causal inference, economic modeling, and machine learning to drive measurable business impact. We are looking for an Applied Science Manager to lead our Ads Impact initiative. This team owns the science of understanding and optimizing how advertising creates value for shoppers and selling partners. What makes this role distinctive is its position at the frontier of AI and Economics: as Amazon's shopping experience evolves from traditional search toward LLM-powered, agentic commerce, the fundamental mechanisms through which advertising creates value are changing. This role will partner with leading scientists and academic researchers to measure these effects through large-scale causal experimentation, and develop novel methods to encode causal and economic reasoning into AI systems that optimize the shopping experience. Key job responsibilities In this role, you will lead a team of scientists, setting the technical vision and science roadmap for ads impact measurement and optimization. You will design experiments that identify the causal mechanisms through which advertising drives shopper engagement, advertiser value, and marketplace outcomes. You will develop optimization algorithms that integrate these causal signals into production and business decision-making, in close partnership with engineering and product teams across the organization. You will lead the research and communicate findings and recommendations to senior leadership through written narratives that connect technical science to business strategy. This role requires deep expertise in causal inference and experimental design, combined with strong applied ML skills and the engineering judgment to translate research into production systems. You will hire and develop future science leaders, think strategically, set ambitious roadmaps in highly ambiguous problem spaces, and foster a culture that values both intellectual depth and production impact. You will work cross-functionally, influencing across organizational boundaries to drive alignment on complex, multi-sided tradeoffs.
Senior Applied Scientist, Industrial Robotics Group
US, WA, Seattle
Amazon Industrial Robotics is seeking exceptional applied science talent to develop AI and machine learning systems that will enable the next generation of advanced manufacturing capabilities at unprecedented scale. We're building revolutionary software infrastructure that combines cutting-edge AI, large-scale optimization, and advanced manufacturing processes to create adaptive production control systems. As a Senior Applied Scientist, you will develop and improve machine learning systems that enable real-time manufacturing flow decisions. You will leverage state-of-the-art optimization and ML techniques, evaluate them against representative manufacturing scenarios, and adapt them to meet the robustness, reliability, and performance needs of production environments. You will invent new algorithms where gaps exist. You'll collaborate closely with software engineering, manufacturing engineering, robotics simulation, and operations teams, and your outputs will directly power the systems that determine what to build next, where to allocate resources, and how to maximize throughput. The ideal candidate brings deep expertise in optimization and machine learning, with a proven track record of delivering scientifically complex solutions into production. You are hands-on, writing significant portions of critical-path scientific code while driving your team's scientific agenda. If you're passionate about inventing the intelligent manufacturing systems of tomorrow rather than optimizing those of today, this role offers the chance to make a lasting impact on the future of automation. Key job responsibilities - Identify and devise new scientific approaches for constraint identification, dispatch optimization, WIP release control, and predictive flow intelligence when the problem is ill-defined and new methodologies need to be invented - Lead the design, implementation, and successful delivery of scientifically complex solutions for real-time manufacturing flow optimization in production - Design and build ML models and optimization algorithms including constraint prediction, starvation risk forecasting, and dispatch optimization - Write a significant portion of critical-path scientific code with solutions that are inventive, maintainable, scalable, and extensible - Execute rapid, rigorous experimentation with reproducible results, closing the gap between simulation and real manufacturing environments - Build evaluation benchmarks that measure model performance against manufacturing outcomes including constraint utilization and throughput rather than traditional ML metrics alone - Influence your team's science and business strategy through insightful contributions to roadmaps, goals, and priorities - Partner with manufacturing engineering, robotics simulation, and applied intelligence teams to ensure scientific approaches are grounded in operational reality - Drive your team's scientific agenda and role model publishing of research results at peer-reviewed venues when appropriate and not precluded by business considerations - Actively participate in hiring and mentor other scientists, improving their skills and ability to deliver - Write clear narratives and documentation describing scientific solutions and design choices
Senior Research Scientist, Ad Measurements Science
US, NY, New York
The Ads Measurement Science team in the Measurement, Ad Tech, and Data Science (MADS) team of Amazon Ads serves a centralized role developing solutions for a multitude of performance measurement products. We create solutions which measure the comprehensive impact of advertiser's ad spend, including sales impacts both online and offline and across timescales, and provide actionable insights that enable our advertisers to optimize their media portfolios. We also own the science solutions for AI tools that unlock new insights and automate high-effort customer workflows, such as custom query and report generation based on natural language user requests. We leverage a host of scientific technologies to accomplish this mission, including Generative AI, classical ML, Causal Inference, Natural Language Processing, and Computer Vision. As a Senior Research Scientist on the team, you will be at the forefront of innovation, developing measurement solutions end-to-end from inception to production. You will set the technical vision and innovate on behalf of our customers. You will propose, design, analyze, and productionize models to provide novel measurement insights to our customers. You will partner with engineering to deploy these solutions into production. You will work with key stakeholders from various business teams to enable advertisers to act upon those metrics. Key job responsibilities * Lead the development of ad measurement models and solutions that address the full spectrum of an advertiser's investment, focusing on scalable and efficient methodologies. * Collaborate closely with cross-functional teams including engineering, product management, and business teams to define and implement measurement solutions. * Use state-of-the-art scientific technologies including Generative AI, Classical Machine Learning, Causal Inference, Natural Language Processing, and Computer Vision to develop state of the art models that measure the impact of ad spend across multiple platforms and timescales. * Drive experimentation and the continuous improvement of ML models through iterative development, testing, and optimization. * Translate complex scientific challenges into clear and impactful solutions for business stakeholders. * Mentor and guide junior scientists, fostering a collaborative and high-performing team culture. * Foster collaborations between scientists to move faster, with broader impact. * Regularly engage with the broader scientific community with presentations, publications, and patents. A day in the life You will solve real-world problems by getting and analyzing large amounts of data, generate business insights and opportunities, design simulations and experiments, and develop statistical and ML models. The team is driven by business needs, which requires collaboration with other Scientists, Engineers, and Product Managers across the advertising organization. You will prepare written and verbal presentations to share insights to audiences of varying levels of technical sophistication. Team video https://advertising.amazon.com/help/G4LNN5YWHP6SM9TJ About the team We are a team of scientists across Applied, Research, Data Science and Economist disciplines. You will work with colleagues with deep expertise in ML, NLP, CV, Gen AI, and Causal Inference with a diverse range of backgrounds. We partner closely with top-notch engineers, product managers, sales leaders, and other scientists with expertise in the ads industry and on building scalable modeling and software solutions.
Applied Scientist I, Buyer Risk Prevention (BRP)
IN, KA, Bengaluru
Do you want to join an innovative team of scientists applying machine learning and advanced statistical techniques to protect Amazon customers and enable a trusted eCommerce experience? Are you excited about modeling terabytes of data and building state-of-the-art algorithms to solve complex, real-world fraud and risk challenges? Do you enjoy owning end-to-end machine learning problems, directly influencing customer experience and company profitability, while collaborating in a diverse, high-performing team? If so, the Amazon Buyer Risk Prevention (BRP) Machine Learning team may be the right fit for you. We are seeking an Applied Scientist to design, develop, and deploy advanced algorithmic systems that safeguard millions of transactions every day. In this role, you will independently drive model development from problem formulation to production deployment, build scalable ML solutions, and leverage emerging technologies—including Generative AI and LLMs—to enhance fraud detection and next-generation risk prevention systems. Key job responsibilities Own end-to-end development of machine learning models for large-scale risk management systems Analyze large volumes of historical and real-time data to identify fraud patterns and emerging risk trends Design, develop, validate, and deploy innovative models to production environments Apply GenAI/LLM technologies to automate risk evaluation and improve operational efficiency Collaborate closely with software engineering teams to implement scalable, real-time model solutions Partner with operations and business stakeholders to translate risk insights into measurable impact Establish scalable and automated processes for data analysis, model experimentation, validation, and monitoring Track model performance and business metrics; communicate insights clearly to technical and non-technical stakeholders Research and implement novel machine learning and statistical methodologies
Applied Scientist
DE, BE, Berlin
At Audible, we believe stories have the power to transform lives. It’s why we work with some of the world’s leading creators to produce and share audio storytelling with our millions of global listeners. We are dreamers and inventors who come from a wide range of backgrounds and experiences to empower and inspire each other. Imagine your future with us. ABOUT THIS ROLE As an Applied Scientist, you will solve large complex real-world problems at scale, draw inspiration from the latest science and technology to empower undefined/untapped business use cases, delve into customer requirements, collaborate with tech and product teams on design, and create production-ready models that span various domains, including Machine Learning (ML), Artificial Intelligence (AI) and Generative AI, Natural Language Processing (NLP), Reinforcement Learning (RL), real-time and distributed systems. ABOUT YOU Your work will focus on inventing or adapting scientific approaches, models, and algorithms driven by customer needs at the project level. You will develop components and/or end-to-end solutions that are deployed into production or directly support production systems, delivering consistently high-quality work that meets both scientific and engineering best practices. You will develop reusable science components and services that resolve architecture deficiencies and customers’ pain points, while making technical trade-offs for long-term/short- term. You will work semi-autonomously to deliver solutions, contribute to research papers at peer-reviewed venues when appropriate, and document your work thoroughly to enable others to understand and reproduce it. Your decision-making will consistently incorporate robust, data-driven business and technical judgment. You will collaborate with other scientists to raise the bar of both scientific and engineering complexity for the team and to foster valuable scientific partnership opportunities to help/guide science decisions. We work in a highly collaborative, fast-paced environment where scientists, engineers, and product managers work to test and build scalable foundational capabilities, as well as customer facing experiences. You will have the opportunity to innovate and think big within your projects scope, implement optimization services and algorithms, and influence the experiences of millions of customers. We are looking for a results-oriented Applied Scientist with deep knowledge in ML, NLP, Deep Learning, GenAI, and/or large-scale distributed computation. As an Applied Scientist, you will... - Understand use cases across the business and adopt/extend/design/invent solutions/models that are scalable, efficient, and automated for difficult problems that are not well defined - Work closely with fellow scientists and software engineers (at Audible and Amazon) to build and productionize models, deliver novel and highly impactful features - Review models of peers for the purpose of reducing and managing risk to the business, while improving customer experience - Design, develop, and deploy modeling techniques and solutions for Content Understanding, Recommendations, GenAI-based product features, by employing a wide range of methodologies, working from simple to complex - Contribute to initiatives that employ the most recent advances in ML/AI in a fast-paced, experimental environment - Push the boundary of innovation ABOUT AUDIBLE Audible is the leading producer and provider of audio storytelling. We spark listeners’ imaginations, offering immersive, cinematic experiences full of inspiration and insight to enrich our customers daily lives. We are a global company with an entrepreneurial spirit. We are dreamers and inventors who are passionate about the positive impact Audible can make for our customers and our neighbors. This spirit courses throughout Audible, supporting a culture of creativity and inclusion built on our People Principles and our mission to build more equitable communities in the cities we call home.
Applied Scientist II, On Road Experience
IN, KA, Bengaluru
This position is based in Bangalore, India The Last Mile team helps get packages from delivery stations to a customer’s doorstep. To provide new innovations for customers, awe are inventing the next-generation smart delivery operation. We are combining innovative mobile and IoT technologies, data streams (video, vehicle telematics, location, and presence), together with machine learning models and algorithms – all to create solutions that allow us to deliver faster, and with more confidence. Playing a key role in the Last Mile Driver Experience team, as a Applied Scientist you will be responsible for building machine learning models and algorithms in areas including mapping and location, pattern detection in sensor data, and computer vision. Using your research, you will work with your engineering and product management peers to drive designs from ideation through development and into production. You will bring your experience of research for similar products and solutions, preferably in consumer or industrial verticals. This role requires autonomy and an ability to deliver results, often within the ambiguity of building a v1 product. You will need to work efficiently to build the right things with limited guidance, raising the bar to create an amazing experience for our customers.
See more jobs
View from space of a connected network around planet Earth representing the Internet of Things.
Get more from Amazon Science
Subscribe to our newsletter

Amazon.com | Conditions of Use | Privacy | © 1996-2026 Amazon.com, Inc. or its affiliates

Social