عرض بسيط للتسجيلة

المؤلفGoyal, Tanya
المؤلفMcDonnell, Tyler
المؤلفKutlu, Mucahid
المؤلفElsayed, Tamer
المؤلفLease, Matthew
تاريخ الإتاحة2024-02-21T08:13:41Z
تاريخ النشر2018-06-15
اسم المنشورProceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018
المعرّفhttp://dx.doi.org/10.1609/hcomp.v6i1.13331
الاقتباسGoyal, T., McDonnell, T., Kutlu, M., Elsayed, T., & Lease, M. (2018, June). Your behavior signals your reliability: Modeling crowd behavioral traces to ensure quality relevance annotations. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 6, pp. 41-49).
الرقم المعياري الدولي للكتاب978-157735799-5
معرّف المصادر الموحدhttps://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85051529515&origin=inward
معرّف المصادر الموحدhttp://hdl.handle.net/10576/52016
الملخصWhile peer-agreement and gold checks are well-established methods for ensuring quality in crowdsourced data collection, we explore a relatively new direction for quality control: estimating work quality directly from workers' behavioral traces collected during annotation. We propose three behavior-based models to predict label correctness and worker accuracy, then further apply model predictions to label aggregation and optimization of label collection. As part of this work, we collect and share a new Mechanical Turk dataset of behavioral signals judging the relevance of search results. Results show that behavioral data can be effectively used to predict work quality, which could be especially useful with single labeling or in a cold start scenario in which individuals' prior work history is unavailable. We further show improvement in label aggregation and reducing labeling cost while ensuring data quality.
راعي المشروعWe thank the many talented crowd members who contributed to our study, and the reviewers for their valuable feedback. This work was made possible by NPRP grant# NPRP 7-1313-1-245 from the Qatar National Research Fund (a member of Qatar Foundation).
اللغةen
الناشرAAAI Press
الموضوعForecasting
Artificial intelligence
العنوانYour Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations
النوعConference Paper
dc.accessType Full Text


الملفات في هذه التسجيلة

Thumbnail

هذه التسجيلة تظهر في المجموعات التالية

عرض بسيط للتسجيلة