Deep learning-based automatic analysis of legal contracts: a named entity recognition benchmark

Abstract

Entity recognition and extraction from contracts play a crucial role in automating contract analysis and extracting valuable information. Named Entity Recognition (NER) techniques are used for identifying and classifying specific entities such as parties, dates, amounts, and clauses within contracts. In this study, we create a high-quality NER dataset from various types of English language contracts by considering their structure, and the legal terminology used within these documents. We present a systematic approach to manually annotate contracts with appropriate entity labels, ensuring accuracy and consistency. The resulting NER dataset serves as a valuable resource for training and evaluating NER models for contract analysis tasks. We evaluate the performance of NER on this dataset using a range of methods. These methods include Conditional Random Fields, various Bidirectional LSTM configurations, and BERT models. Each of these models brings different strengths and capabilities to the task of entity recognition, allowing for a comprehensive evaluation and the selection of the best models over the dataset. Among these, the NER model based on Contracts-BERT-base from the Legal-BERT family, which is pre-trained specifically on English contracts, outperformed all others, achieving an impressive overall F1 score of 0.94.