Machine learning (ML) has become a central component in modern software applications, giving rise to many new challenges [8, 15, 20]. Tremendous progress has been made in this context with respect to model serving [1, 6, 10], experiment tracking [14, 16, 22, 23], model diagnosis [4, 5, 11, 21] and data validation [4, 18]. In this paper, we focus on the arising challenge of automating the operation of deployed