Show simple item record

AuthorGoyal, Tanya
AuthorMcDonnell, Tyler
AuthorKutlu, Mucahid
AuthorElsayed, Tamer
AuthorLease, Matthew
Available date2024-02-21T08:13:41Z
Publication Date2018-06-15
Publication NameProceedings of the 6th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018
Identifierhttp://dx.doi.org/10.1609/hcomp.v6i1.13331
CitationGoyal, T., McDonnell, T., Kutlu, M., Elsayed, T., & Lease, M. (2018, June). Your behavior signals your reliability: Modeling crowd behavioral traces to ensure quality relevance annotations. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 6, pp. 41-49).
ISSN978-157735799-5
URIhttps://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85051529515&origin=inward
URIhttp://hdl.handle.net/10576/52016
AbstractWhile peer-agreement and gold checks are well-established methods for ensuring quality in crowdsourced data collection, we explore a relatively new direction for quality control: estimating work quality directly from workers' behavioral traces collected during annotation. We propose three behavior-based models to predict label correctness and worker accuracy, then further apply model predictions to label aggregation and optimization of label collection. As part of this work, we collect and share a new Mechanical Turk dataset of behavioral signals judging the relevance of search results. Results show that behavioral data can be effectively used to predict work quality, which could be especially useful with single labeling or in a cold start scenario in which individuals' prior work history is unavailable. We further show improvement in label aggregation and reducing labeling cost while ensuring data quality.
SponsorWe thank the many talented crowd members who contributed to our study, and the reviewers for their valuable feedback. This work was made possible by NPRP grant# NPRP 7-1313-1-245 from the Qatar National Research Fund (a member of Qatar Foundation).
Languageen
PublisherAAAI Press
SubjectForecasting
Artificial intelligence
TitleYour Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations
TypeConference Paper
dc.accessType Full Text


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record