Optimal Clustering from Noisy Binary Feedback.

We study the problem of clustering a set of items from binary user feedback.Such a problem arises in crowdsourcing platforms solving large-scale labelingtasks with minimal effort put on the users. For example, in some of the recentreCAPTCHA systems, users clicks (binary answers) can be used to efficientlylabel images. In our inference problem, items are grouped into initiallyunknown non-overlapping clusters. To recover these clusters, the learnersequentially presents to users a finite list of items together with a questionwith a binary answer selected from a fixed finite set. For each of these items,the user provides a noisy answer whose expectation is determined by the itemcluster and the question and by an item-specific parameter characterizing thehardness of classifying the item. The objective is to devise an algorithmwith a minimal cluster recovery error rate. We derive problem-specificinformation-theoretical lower bounds on the error rate satisfied by anyalgorithm, for both uniform and adaptive (list, question) selection strategies.For uniform selection, we present a simple algorithm built upon the K-meansalgorithm and whose performance almost matches the fundamental limits. Foradaptive selection, we develop an adaptive algorithm that is inspired by thederivation of the information-theoretical error lower bounds, and in turnallocates the budget in an efficient way. The algorithm learns to select itemshard to cluster and relevant questions more often. We compare the performanceof our algorithms with or without the adaptive selection strategy numericallyand illustrate the gain achieved by being adaptive.