One way to make Amazon’s product discovery algorithms more satisfying is to ensure that we match high-quality products to customer queries. Recently, we conducted a study in which we used crowdsourcing to identify product data that could help predict human judgments of quality; we were scheduled to report the results of that study next week at the ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR), but unfortunately, the conference has been canceled.
In the study, we presented crowd workers with images of pairs of related products — together with public product information supplied by both sellers and customers — and asked them which products were of higher quality. Then we asked them which terms extracted from the product information best explained their judgments.
Our first finding was that perceived quality is not a good predictor of customers’ purchase decisions. We suspect, however, that this is related to our second finding: perceived quality is highly correlated with price. Customers may choose lower-quality products if they’re correspondingly lower priced.
If this hypothesis is right, then a good product discovery algorithm should factor in indications of quality that are not correlated with price. Where should it look for them?
We found that most of the time, the terms that best described the crowd workers’ judgment criteria came from the public customer-supplied information, not the seller information. The customer-supplied information had two sources: customer reviews and question-and-answer sequences in which customers answered other customers’ product-related questions.
In our study, each product pair included one product that was actually purchased and one that was clicked but not purchased during the same query. Products also had to share the most fine-grained classification available in the amazon.com product classification hierarchy. The terms presented to the crowd workers were chosen based on how frequently they appeared in texts associated with those fine-grained categories.
Existing research on product recommendation has mainly focused on modeling purchases directly, without attempting to find the reasons behind customer decisions. We believe that understanding the processes that underlie customers' purchasing decisions will help us make better product recommendations. This work represents one of several steps we’re taking in that direction.
