Effective Recommendation Of Reviewers For Research Proposals

Abstract

In this project, we address the problem that a research funding agency may face when matching potential reviewers with submitted research proposals. A list of potential reviewers for a given proposal is typically selected manually by a small technical group of individuals in the agency. However, the manual approach can be an exhausting and challenging task, and (more importantly) might lead to ineffective selections that affect the subsequent funding decisions. This research work presents an effective automated system that recommends reviewers for proposals and helps program managers in the assignment process. This system views the CVs of the reviewers and rank them by assigning weights for each CV against the list of all the proposals. We propose an automatic method to effectively recommend (for a given research proposal) a short list of potential reviewers who demonstrate expertise in the given research field/topic. To accomplish this task, our system extracts information from the full-text of proposals and the CVs of reviewers. We discuss the proposed solution, and the experience in using the solution within the workflow of the Qatar National Research Fund (QNRF). We evaluate our system on a QNRF/NPRP dataset that includes the submitted proposals and approved list of reviewers from the first 5 cycles of the NPRP funding program. Experimental results on this dataset validate the effectiveness of the proposed approach, and show that the best performance of our system demonstrated for proposals in three research areas: natural science, engineering, and medical. The system does not perform as well for proposals in the other two domains, i.e., humanities and social sciences. Our approach performs very well in overall evaluation with 68% of relevant results, i.e., from each 10 recommendations 7 are matching perfectly. Our proposed approach is general and flexible. Variations of the approach can be used in other applications such as conference paper assignment to reviewers and teacher-course assignment. Our research demonstrates also that there are significant advantages to applying recommender system concepts to the proposal-to-reviewer assignment problem. In summary, the problem of automatic assignment of proposals to reviewers is challenging and time-consuming when it is conducted manually by the program managers. Software systems can offer automated tools that significantly facilitate the role of program managers. We follow previous approaches in treating reviewers finding system as an information retrieval task. We use the same basic tools but the goal is to find relevant people rather than searching for relevant documents. For a specific user query (proposal), the system returns a list of qualified reviewers, ranked by their relevance to the query.