MACE

When evaluating redundant annotations (like those from Amazon’s MechanicalTurk), we usually want to

1. aggregate annotations to recover the most likely answer

2. find out which annotators are trustworthy

3. evaluate item and task difficulty

MACE solves all of these problems, by learning competence estimates for each annotators and computing the most likely answer based on those competences.