When rank order isn't enough: New statistical-significance-aware correlation measures
المؤلف | Kutlu M. |
المؤلف | Elsayed T. |
المؤلف | Hasanain M. |
المؤلف | Lease M. |
تاريخ الإتاحة | 2020-02-24T08:57:14Z |
تاريخ النشر | 2018 |
اسم المنشور | International Conference on Information and Knowledge Management, Proceedings |
المصدر | Scopus |
الملخص | Because it is expensive to construct test collections for Cranfield-based evaluation of information retrieval systems, a variety of lower-cost methods have been proposed. The reliability of these methods is often validated by measuring rank correlation (e.g., Kendall's t) between known system rankings on the full test collection vs. observed system rankings on the lower-cost one. However, existing rank correlation measures do not consider the statistical significance of score differences between systems in the observed rankings. To address this, we propose two statistical-significance-aware rank correlation measures, one of which is a head-weighted version of the other. We first show empirical differences between our proposed measures and existing ones. We then compare the measures while benchmarking four system evaluation methods: pooling, crowdsourcing, evaluation with incomplete judgments, and automatic system ranking. We show that use of our measures can lead to different experimental conclusions regarding reliability of alternative low-cost evaluation methods. |
راعي المشروع | This work was made possible by NPRP grant# NPRP 7-1313-1-245 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors. |
اللغة | en |
الناشر | Association for Computing Machinery |
الموضوع | Evaluation IR System Ranking Rank Correlation |
النوع | Conference Paper |
الصفحات | 397 - 406 |
الملفات في هذه التسجيلة
الملفات | الحجم | الصيغة | العرض |
---|---|---|---|
لا توجد ملفات لها صلة بهذه التسجيلة. |
هذه التسجيلة تظهر في المجموعات التالية
-
علوم وهندسة الحاسب [2284 items ]