عرض بسيط للتسجيلة

المؤلفRahman, Md Mustafizur
المؤلفKutlu, Mucahid
المؤلفElsayed, Tamer
المؤلفLease, Matthew
تاريخ الإتاحة2024-02-21T05:51:44Z
تاريخ النشر2020-09-14
اسم المنشورICTIR 2020 - Proceedings of the 2020 ACM SIGIR International Conference on Theory of Information Retrieval
المعرّفhttp://dx.doi.org/10.1145/3409256.3409837
الاقتباسRahman, M. M., Kutlu, M., Elsayed, T., & Lease, M. (2020, September). Efficient test collection construction via active learning. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval (pp. 177-184).
الرقم المعياري الدولي للكتاب978-145038067-6
معرّف المصادر الموحدhttps://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85093118866&origin=inward
معرّف المصادر الموحدhttp://hdl.handle.net/10576/51995
الملخصTo create a new IR test collection at low cost, it is valuable to carefully select which documents merit human relevance judgments. Shared task campaigns such as NIST TREC pool document rankings from many participating systems (and often interactive runs as well) in order to identify the most likely relevant documents for human judging. However, if one's primary goal is merely to build a test collection, it would be useful to be able to do so without needing to run an entire shared task. Toward this end, we investigate multiple active learning strategies which, without reliance on system rankings: 1) select which documents human assessors should judge; and 2) automatically classify the relevance of additional unjudged documents. To assess our approach, we report experiments on five TREC collections with varying scarcity of relevant documents. We report labeling accuracy achieved, as well as rank correlation when evaluating participant systems based upon these labels vs. full pool judgments. Results show the effectiveness of our approach, and we further analyze how varying relevance scarcity across collections impacts our findings. To support reproducibility and follow-on work, we have shared our code online\footnote\urlhttps://github.com/mdmustafizurrahman/ICTIR_AL_TestCollection_2020/.
راعي المشروعQatar National Research Fund (QNRF) - NPRP 7-1313-1-245.
اللغةen
الناشرAssociation for Computing Machinery
الموضوعactive learning
evaluation
information retrieval
test collections
العنوانEfficient Test Collection Construction via Active Learning
النوعConference Paper
الصفحات177–184
dc.accessType Open Access


الملفات في هذه التسجيلة

Thumbnail

هذه التسجيلة تظهر في المجموعات التالية

عرض بسيط للتسجيلة