Leveraging confidence models for identifying challenging data subgroups in speech models

State-of-the-art speech models may exhibit suboptimal performance in specific population subgroups. Detecting these challenging subgroups is crucial to enhance model robustness and fairness. Traditional methods for subgroup identification typically rely on demographic information such as age, gender, and origin. However, collecting such sensitive data at deployment time can be impractical or unfeasible due to privacy concerns.

This paper introduces a novel Challenging Subgroup Identification model (CSI) to (i) automatically predict if an utterance belongs to a challenging subgroup and (ii) provide an interpretable representation of this subgroup. CSI exploits confidence models (CMs) to encode information about sources of errors, as CMs assess model certainty of predictions, providing insights into output reliability. CM finetuning based on challenging subgroup identification techniques allows accurate subgroup identification. CSI leverages demographic features only during its training, avoiding the need for sensitive data collection at deployment time. Experimental results on the automatic speech recognition and intent classification tasks show CSI effectiveness in identifying challenging subgroups and providing interpretable subgroup descriptions. These findings highlight CSI as a valuable tool for improving the robustness and fairness of speech models in real-world applications.