Human Expertise Drives Machine Learning for Contract Review

Introduction:

Contract review powered by AI technology still relies on expert human review for training and fine-tuning the ML algorithms. Cenza’s expertise in annotating and tagging meta-data is an essential part of the real-world requirements of companies today as they seek to deploy AI/ML technology to tackle large volumes of unstructured contract data.

The Client:

A US-based company that provides an AI-powered contract management and analytics tool for in-house legal and finance teams. The company’s platform automatically finds and extracts key terms and meta-data from contracts and provides key insights with deep search, custom reporting, and analytics, thus enabling clients to track contracts and cut costs by eliminating manual reviews.

The Challenge:

When it comes to developing algorithms to build accurate machine learning models, there needs to be a very low margin of error. Solutions that support contract management are no different in terms of accuracy and proficiency needs due to the large volume of data as well as the wide variety of contract types.

Training the client’s AI-powered tool for contract review required expert human reviewers to manually annotate and tag huge volumes of contract data. There was also a tight execution timeline tied to the planned rollout of this new platform. From the client’s perspective, this was a compelling case to pursue an outsourcing solution. However, this would only be feasible if an expert contract review team could be assembled quickly and could perform with sufficient care and expertise to ensure that the data would be annotated and tagged with the utmost accuracy to satisfy the high standards of the client’s Data Science & Product team.

The Solution:

Cenza’s contract review teams comprise qualified and extremely well-trained attorneys with diverse legal backgrounds and extensive experience in all phases of contract management, including drafting, review, abstraction, redlining, and summarization. Our ability to quickly bring together a highly skilled review team with expertise that spans multiple legal domains was key to the success of this project.

• Our first task was to assemble an initial review team consisting of 10 attorneys, with domain expertise broad enough to review and annotate a wide array of corporate and commercial contracts, including NDAs, MSAs, sales and employment agreements, among others. This team also included 5 Quality Specialists and a Project Manager.
• Next, we established a secure work environment for the review team in a dedicated and locked down area of our facility. This allowed the team to operate within the client’s firewall and maintain the highest level of data security, while using the client’s AI-powered contract management platform to perform all phases of the assignment.
• Before commencing review, we converted the contracts into machine readable format, first with OCR scanning, followed by careful proofreading in order to maintain the level of accuracy necessary for system training.
• Once all the data had been converted, our initial review team performed its abstraction, annotation, and tagging of key terms and contract meta-data, including parties, effective dates, payments terms and dates, as well as termination and auto-renewal provisions. To facilitate system training, the human review team highlighted and linked contract provisions to the appropriate data field, with a focus on named entity recognition values (NER) for each type of contract in order to properly train the AI algorithm. The abstraction covered the 13 standard data fields and extensive custom fields, specific to each contract type and client specifications.

Cenza’s Quality Specialists undertook an independent review of the tagging and data abstraction in order to ensure the team was capturing the appropriate NER values and performing in accordance with the client’s rigorous requirements.

“Thanks for the effort you’ve put forth to help us train our algorithm. This work requires an understanding of legal verbiage and attention to detail. Cenza’s team clearly possesses these skills. Machine Learning has grown in leaps and bounds since we can feed in massive amounts of correct data. The team at Cenza is diligent, flexible, and delivers quality work within the deadlines.”

Director of Legal, a US-based AI-powered contract management and analytics provider

The Overall Impact:

1. Through seamless integration with the client’s AI platform, the Cenza team completed multiple layers of review on a diverse set of over 100,000 contracts and annotated more than 1 million key terms within a three-month timeframe. Accuracy exceeded a 99% rate threshold, thanks to our rigorous focus on quality and project management.
2. Cenza provides a fully scalable, high-quality, and completely secure outsourced solution for contract review that can be customized to the needs of any client’s AI/ML system training requirements. No matter what type of contracts and unstructured data needs to be process, we will assemble an expert review team that has the domain expertise matched to your needs. This means we will be able to keep your project on track, on time, under budget, and exceeding your requirements for machine learning accuracy.

We serve AI/ML clients across the LegalTech, RegTech, and PropTech space, all of whom are involved in all stages of contract review, deal with multiple contract types, and across different industries. The one thing our clients all have in common is that they understand first-hand the value Cenza delivers in successfully handling their AI/ML training assignments.