عرض بسيط للتسجيلة

المؤلفMcDonnell, Tyler
المؤلفLease, Matthew
المؤلفKutlu, Mucahid
المؤلفElsayed, Tamer
تاريخ الإتاحة2024-02-21T09:07:03Z
تاريخ النشر2016-09
اسم المنشورProceedings of the AAAI Conference on Human Computation and Crowdsourcing
المعرّفhttp://dx.doi.org/10.1609/hcomp.v4i1.13287
الاقتباسMcDonnell, T., Lease, M., Kutlu, M., & Elsayed, T. (2016, September). Why is that relevant? collecting annotator rationales for relevance judgments. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 4, pp. 139-148).
معرّف المصادر الموحدhttp://hdl.handle.net/10576/52019
الملخصWhen collecting subjective human ratings of items, it can be difficult to measure and enforce data quality due to task subjectivity and lack of insight into how judges’ arrive at each rating decision. To address this, we propose requiring judges to provide a specific type of rationale underlying each rating decision. We evaluate this approach in the domain of Information Retrieval, where human judges rate the relevance of Webpages to search queries. Cost-benefit analysis over 10,000 judgments collected on Mechanical Turk suggests a win-win: experienced crowd workers provide rationales with almost no increase in task completion time while providing a multitude of further benefits, including more reliable judgments and greater transparency for evaluating both human raters and their judgments. Further benefits include reduced need for expert gold, the opportunity for dual-supervision from ratings and rationales, and added value from the rationales themselves.
اللغةen
الناشرAAAI Press
الموضوعrelevance judgments
task design
annotator agreement
rationale task
gold standard
standard task
stage task
pilot study
annotator rationale
experienced worker
data quality
mechanical turk
dual supervision
العنوانWhy Is That Relevant? Collecting Annotator Rationales for Relevance Judgments
النوعConference Paper
الصفحات139-148
رقم العدد1
رقم المجلد4
dc.accessType Full Text


الملفات في هذه التسجيلة

Thumbnail

هذه التسجيلة تظهر في المجموعات التالية

عرض بسيط للتسجيلة