Individual Query Cardinality Estimation using Multiple Query Combinations on a Search Engine's Corpus

Abstract

Most modern search engines feature keyword based search interfaces. These interfaces are usually found on websites belonging to enterprises or governments or sites related to news articles, blogs and social media that contain a large corpus of documents. These collections of documents are not easily indexed by web search engines, and are considered as hidden web databases. These databases provide opportunities for data analysis for many third-parties through their keyword search interfaces. A significant amount of research has already been carried out on analyzing and extracting aggregate information about these hidden document corpora. But most of these research focus on the high level big-picture information of the database. Not enough focus has been done on extracting analytical information which is specific to individual queries. This paper focuses on that analysis gap and takes ideas from other existing research to formulate a query cardinality estimation technique i.e. The count of documents matching a query in the document corpus of a search engine. We experimentally assess the effectiveness of our method by building a search engine on the Reuters-21578 document corpus. For a given keyword the corresponding documents' count is estimated only by sending search queries using the interface. 1 2017 IEEE.