RETRIEVAL OF AUTHORITIES AND THEIR EVIDENCE FOR RUMOR VERIFICATION IN ARABIC SOCIAL MEDIA

Abstract

Social media platforms have become a medium for rapidly spreading rumors along with emerging events. Those rumors may have a lasting effect on users' opinion even after it is debunked, and may continue to influence them if not replaced with convincing evidence. Journalists, or even normal users, who attempt to verify a rumor over social media, try to find a trusted source of evidence that can help them confirm or deny that specific rumor. A strong source of evidence for verifying a rumor is an authority who has the "real knowledge or power" to verify it if asked to. This dissertation contributes towards addressing the problem of rumor verification in social media. We propose augmenting the traditional rumor verification pipeline, which considers the propagation networks and the Web as sources of evidence, by incorporating authorities as another source of evidence. Specifically, in this dissertationwe introduce the problem of rumor verification using evidence from authorities which we believe can help fact-checkers and automated rumor verification systems to find the right authorities and evidence from their Twitter timelines, hence helping in the verification process. First, we propose authority finding in Twitter. We then suggest incorporating those retrieved authorities by detecting their stance towards rumors in Twitter, and retrieving evidence from their timeline tweets. Finally, we propose rumor verification using evidence retrieved from those authorities. To address the problem, we construct and release three datasets targeting the Arabic language namely 1) the first Authority FINding in Twitter (AuFIN) which comprises 150 rumors (expressed in tweets) associated with a total of 1,044 authority accounts and a user collection of 395,231 Twitter accounts (members of 1,192,284 unique Twitter lists), 2) the first Authority STance towards Rumors (AuSTR) which comprises 811 (rumor tweet, authority tweet) pairs relevant to 292 unique rumors, 3) the first Authority- Rumor-Evidence Dataset (AuRED) which comprises 160 rumors expressed in tweets and 692 Twitter timelines of authorities comprising about 34k annotated tweets in total. We propose a hybrid retrieval authority finding model that combines lexical and semantic signals in addition to user profiles and network features. Furthermore, we investigate the usefulness of existing Arabic datasets for stance towards claims for detecting the stance of authorities. Finally, we study the effectiveness of existing factchecking models for evidence retrieval from authorities and rumor verification using the retrieved evidence. Our experimental results suggest that Twitter lists and network features such as followers, and followees count, adopted previously for topic expert finding models, play a crucial role in authority finding; however, they are insufficient. This motivates the need to explore other features to differentiate experts from authorities. Moreover, our proposed hybrid model incorporating lexical, semantic, and user network features achieved a modest performance, 0.41 as precision at depth 1, which indicates that finding authorities is a challenging task, and that there is still room for continued enhancement. Our results also highlighted that adopting existing Arabic stance datasets for claim verification is somewhat useful but clearly insufficient for detecting the stance of authorities. Moreover, we found that AuSTR solely, despite the limited size, can be sufficient for detecting the stance of authorities achieving a performance of 0.84 macro-F1 and 0.78 F1 on debunking tweets. Our investigation on the effectiveness of existing fact-checking (claim verification using evidence from Wikipedia pages) models on our problem highlighted that although evidence retrieval for fact-checking models performrelativelywell on evidence retrieval from authorities, establishing strong baselines achieving 0.70 as recall at depth 5, there is still a big room for improvement. However, existing claim verification for fact-checking models perform poorly on rumor verification using evidence from authorities, 0.42 as macro-F1, no matter how good the retrieval performance is. Moreover, existing fact-checking datasets showed a potential in transfer learning to our problem, however, further investigation using different setups and datasets is required. Furthermore, drawing upon our experiments, we discuss failure factors and make recommendations for future research directions in addressing this problem. Additionally, our approach establishes a strong baseline for future studies targeting automatic rumor verification in social media, and our constructed datasets can facilitate further research on the problem. Finally, our proposed system can be integrated into verification systems, and can be also exploited by fact-checkers or journalists to find trusted sources of evidence.