LLM-based playlist curation: A music reasoning approach at scale

We present a production-scale playlist curation system on Amazon Music that leverages modular architecture and holistic LLM-powered reasoning to generate playlists. Our system leverages LLMs to reason over rich track metadata—including genre, mood, era, sonic descriptions, and artist context—to select cohesive track sets that satisfy both relevance and coherence criteria. We formulate the curation task under two complementary inference paradigms: track-wise (binary classification) and list-wise (subset selection). We systematically compare these paradigms and evaluate eight prompt variants across three design dimensions: playlist-level reasoning, track-level reasoning and output structure. Our prompt optimization establishes that outputting only negative tracks achieves 0.96 F1 with 84% fewer output tokens compared to naive all-track approaches. To meet production latency and cost constraints, we develop a teacher-distillation pipeline that transfers reasoning capabilities from Claude Opus 4.6 to Qwen3.5-4B via supervised fine-tuning, achieving comparable playlist quality with 227x cost reduction. Evaluation via recreating both expert-curated editorial playlists and real user personalized playlists demonstrates consistent improvements over production baselines across relevance metrics (weighted precision, weighted NDCG) and coherence metrics (consumption similarity, valence deviation, danceability deviation). Online A/B experiments further validate the framework with improvements in user engagement metric (+0.06%) and user retention metric (+0.03%), confirming that LLMs can serve as effective quality gates for algorithmic playlist curation at scale.