Show simple item record

AuthorMcDonnell, Tyler
AuthorKutlu, Mucahid
AuthorElsayed, Tamer
AuthorLease, Matthew
Available date2024-02-21T08:54:54Z
Publication Date2017-08
Publication NameIJCAI International Joint Conference on Artificial Intelligence
Identifierhttp://dx.doi.org/10.24963/ijcai.2017/692
CitationMcDonnell, T., Kutlu, M., Elsayed, T., & Lease, M. The Many Benefits of Annotator Rationales for Relevance Judgments.
ISBN978-099924110-3
ISSN1045-0823
URIhttps://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85031942270&origin=inward
URIhttp://hdl.handle.net/10576/52018
AbstractWhen collecting subjective human ratings of items, it can be difficult to measure and enforce data quality due to task subjectivity and lack of insight into how judges arrive at each rating decision. To address this, we propose requiring judges to provide a specific type of rationale underlying each rating decision. We evaluate this approach in the domain of Information Retrieval, where human judges rate the relevance of Webpages. Costbenefit analysis over 10,000 judgments collected on Mechanical Turk suggests a win-win: experienced crowd workers provide rationales with no increase in task completion time while providing further benefits, including more reliable judgments and greater transparency.
SponsorThis work was made possible by NPRP grant NPRP 7-1313-1-245 from the Qatar National Research Fund (a member of Qatar Foundation).
Languageen
PublisherInternational Joint Conferences on Artificial Intelligence
SubjectArtificial intelligence
Relevance judgment
TitleThe many benefits of annotator rationales for relevance judgments
TypeConference Paper
Volume Number0
dc.accessType Open Access


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record