MACE

in webapps

November 24, 2020

When evaluating redundant annotations (like those from Amazon’s MechanicalTurk), we usually want to

  1. aggregate annotations to recover the most likely answer

  2. find out which annotators are trustworthy

  3. evaluate item and task difficulty

MACE solves all of these problems, by learning competence estimates for each annotators and computing the most likely answer based on those competences.

Posted on:
November 24, 2020
Length:
1 minute read, 55 words
Categories:
webapps
See Also: