Ranking disagreement between graders: Application to ranking hadith scholars

Abstract

Ranking raters based on their agreement or disagreement is a very active research field with many applications. It has been applied for instance for ranking automatic essay grading algorithms based on their agreement with human graders. Many metrics have been introduced for measuring agreement between graders with different performances depending on the type of data. In this paper, we apply the agreement measures on a new field which is the hadith authentication problem. Indeed, reported hadiths (actions or sayings of Prophet Muhamad PBUH) are not always authentic and scholars who study their authenticity often disagree on some hadiths. In this study, many metrics have been proposed and studied on a simulated data in order to determine which metric best reflects the measure of agreement especially in the presence of missing data. Best performing metric has been applied to real data in order to rank the hadith scholars. This study is a breakthrough in the science of hadith as it makes it possible to assign a confidence score to each authenticity judgment that reflects how often the hadith scholar in question agrees or disagrees with other scholars.