Learning to Classify from Multiple Annotators

In many practical machine learning applications in which data are labeled by human experts or human annotators it is infeasible to assume one can obtain a perfect set of labels everybody agrees on. The labels provided by humans are often subjective and depend on many factors such as their background and knowledge, the amount of time and care they spend on labeling, and the utilities (or costs) they associate with correct and incorrect label assignments. In this paper we study the question of how to learn a good classification model when there are disagreements among labels of multiple annotators. We propose and develop a new framework for learning classification models that explicitly models and learns annotator-specific models, the reliability of different annotators, and incorporates them into the learning of the consensus model. Experimental results on multiple data sets show that our method outperforms existing multiple annotators baselines.