Scalable heterogeneity detection in online experiments

2024
Download Copy BibTeX
Copy BibTeX
Online sites typically evaluate the impact of new product features on customer behavior using online controlled experiments (or A/B tests). For many business applications, it is important to detect heterogeneity in these experiments [1], as new features often have a differential impact by customer segment, product group, and other variables. Understanding heterogeneity can provide key insights into causal pathways, enable differential launches to subgroups of customers and products, and motivate future experiments.

Recent methods have focused on estimating heterogeneous impacts [1], but they are difficult to apply to online experiments for two reasons. First, existing methods scale poorly in industry settings. With hundreds of thousands of experiments and billions of observations per year, many existing methods are impractical due to their computational speed and need for supervision. Second, existing methods typically detect heterogeneity based on customer features. However, heterogeneity also exists on other axes. For example, a new online retail feature may increase e-book revenue, but reduce revenue from physical books, which reflects heterogeneity in a dimension of the outcome (i.e., revenue broken out by product group). Current approaches for such heterogeneity (e.g., fixed effects) scale poorly with large numbers of customers and categories. There is thus a need for a scalable method that can detect different types of heterogeneity in online experiments.
Research areas
Tags

Latest news

GB, MLN, Edinburgh
We’re looking for a Machine Learning Scientist in the Personalization team for our Edinburgh office experienced in generative AI and large models. You will be responsible for developing and disseminating customer-facing personalized recommendation models. This is a hands-on role with global impact working with a team of world-class engineers and scientists across the Edinburgh offices and wider organization. You will lead the design of machine learning models that scale to very large quantities of data, and serve high-scale low-latency recommendations to all customers worldwide. You will embody scientific rigor, designing and executing experiments to demonstrate the technical efficacy and business value of your methods. You will work alongside aRead more