Effectiveness of social media sentiment analysis tools with the support of emoticon/emoji

Abstract

Organizations are increasingly interested in using microblogging platforms, such as Twitter, to get rapid feedback in several domains using sentiment analysis algorithms to rate, for example, whether a target audience is happy or unhappy. However, posts on microblogging platforms can differ from the source material used to train the sentiment analysis tools. For example, emojis and emoticons are increasingly employed in social media to clarify, enhance, or sometimes reverse the sentiment of a post but can be stripped out of a piece of text before it is processed. Responding to this interest, many sentiment analysis algorithms are being made available as web services, but as details of the algorithms used are not always published on the website, comparisons between web services and how well they deal with the peculiarities of microblogging posts can be difficult. To address this, a prototype web application was developed to compare the performance of nine tweet-related sentiment analysis web services and, through targeted hypotheses, to study the effect of emojis and emoticons on polarity classification. Twelve specific research test sets were created with the application, labelled by volunteers, and tested against the analysis web services with evaluation provided by two- and three-class accuracy measures. Distinct differences were found in how the web services used emoticons and emojis in assigning a positive or negative sentiment value to a tweet, with some services seeming to ignore their presence. It was found in general that web services classified polarity sensitive tweets significantly less accurately than tweets where the sentiment of the emoji/emoticon supported the sentiment of the text.