What it takes to build an AI legal assistant lawyers can rely on

AI and machine learning experts, experienced attorneys, thousands of hours of prompt engineering—and that’s just to launch

Since OpenAI released the generative AI application ChatGPT, conversation about this technology’s impact on the legal profession has been nonstop. And speculation only increased when GPT-4, the world’s most advanced large language model (LLM) (which powers the subscription service ChatGPT Plus), was unveiled in March.

As discussed in our first post in this series, many believe it’s too soon for lawyers to rely on ChatGPT or GPT-4 for legal practice because they hallucinate, and because they don’t access up-to-date, accurate legal data on their own. OpenAI itself has cautioned users about relying on GPT-4’s output, especially when stakes are high. 

But it’s not true that lawyers cannot trust generative AI for legal practice. It’s only true that they cannot trust generative AI alone—a crucial distinction. It is possible to build a product integrating GPT-4 that meets professional standards. Which is exactly what we’ve done with CoCounsel. But how?

Channeling GPT-4’s power into a reliable legal AI platform

Though LLMs have only been part of daily news for the last several months, they’ve been in play for the last several years, and our engineers have worked with them since 2018 to create products such as Parallel Search. But the superiority of GPT-4’s reasoning abilities made its release a turning point. No prior model had been capable of performing legal reasoning this well. So why isn’t GPT-4 on its own enough? The chief issue is memory. 

GPT-4 sometimes gives incorrect answers to questions and even fabricates information (hallucinates), because its only source of information is its own memory, which is limited to publicly available information through September 2021. And that public data includes plenty of unreliable information packed with “untruths, hate speech and other garbage.” 

But when GPT-4 is part of an ecosystem—as “brain power” consuming, analyzing, and synthesizing information—that also includes memory comprising not simply publicly available information but domain-specific databases, producing trustworthy output is possible. 

This is why OpenAI selected Casetext to use GPT-4 in building a product suitable for legal professionals. Having led innovation in legal AI since 2013, we have the right “memory,” and the tools to retrieve the right parts of that memory, to anchor GPT-4’s reasoning. The result of this integration? CoCounsel.

AI that’s grounded in the law

In building CoCounsel, Casetext’s product and engineering teams integrated GPT-4 with information it doesn’t have: our legal database, a comprehensive corpus of accurate, up-to-date law such as state and federal case law, statutes, regulations, codes, and rules.

This means all CoCounsel’s output is drawn from a thorough compilation of legal information, because our engineers have also “instructed” the platform to base answers on actual passages contained in the databases or not to answer at all, leaving no opportunity for CoCounsel to hallucinate.

The third element in the CoCounsel ecosystem, in addition to the “brain” (GPT-4) and the “memory” (our databases), is our “appendages,” proprietary tools Parallel Search and AllSearch. These guide GPT-4 to retrieve the right data from memory to answer a user’s legal question, and to do so quickly. 

4,000+ hours of expert prompt engineering

Grounding GPT-4 in the law was only the first step. Next came prompt engineering—another term, like generative AI, that’s now commonplace in even mainstream media. LLM “prompts” are essentially the questions it’s asked, or queries. Effective prompts prevent hallucinations and ensure accurate, complete answers. To create them, “one needs to provide clear and unambiguous language, context and background information, break down complex questions, experiment with different phrasings, and monitor generated content for accuracy and biases.” 

Prompt engineering began with establishing our Trust Team—a group of expert AI engineers and experienced litigators and transactional attorneys—to come up with the “clear and sufficient context” vital to getting the model to generate useful answers. They selected and designed thousands of prompts, and entered them into CoCounsel. 

The team reviewed the prompt output, made slight changes to the prompt content and phrasing to increase the output’s quality and accuracy, then entered it again. After doing this several times for every prompt, they filtered and ranked the answers, selected the best ones, and fed that information back into the CoCounsel. This oversight and refinement is vital to maximizing the value of generative AI. Every bit of this feedback improves results.

Only after more than 4,000 hours of this work, based on about 30,000 legal questions entered into CoCounsel between October 2022 and March 2023, did our team deem the product safe for professional use and ready to launch. 

Continual refinement and expansion of CoCounsel’s skills

Since launching CoCounsel, we’ve continued testing each of its seven skills daily—and launched an eighth—by entering and checking thousands of queries. We’ve also built a backend alert process into the product, in which CoCounsel screens for and flags potential inaccuracies for review, to prevent them from surfacing for end users. 

Perhaps most important, our customer success, product, and engineering teams log and read every single comment and suggestion we get from CoCounsel users. We use this information to make improvements, develop additional skills, and make decisions about what changes and additions to prioritize, based on what our customers want and need most. 

Our next post in this series discusses the importance of customer and client data privacy and security in any AI solution suitable for professional legal use.

Featured posts

Draft Correspondence

Rapidly draft common legal letters and emails.

How this skill works

  • Specify the recipient, topic, and tone of the correspondence you want.

  • CoCounsel will produce a draft.

  • Chat back and forth with CoCounsel to edit the draft.

Review Documents

Get answers to your research questions, with explanations and supporting sources.

How this skill works

  • Enter a question or issue, along with relevant facts such as jurisdiction, area of law, etc.

  • CoCounsel will retrieve relevant legal resources and provide an answer with explanation and supporting sources.

  • Behind the scenes, Conduct Research generates multiple queries using keyword search, terms and connectors, boolean, and Parallel Search to identify the on-point case law, statutes, and regulations, reads and analyzes the search results, and outputs a summary of its findings (i.e. an answer to the question), along with the supporting sources and applicable excerpts.

Legal Research Memo

Get answers to your research questions, with explanations and supporting sources.

How this skill works

  • Enter a question or issue, along with relevant facts such as jurisdiction, area of law, etc.

  • CoCounsel will retrieve relevant legal resources and provide an answer with explanation and supporting sources.

  • Behind the scenes, Conduct Research generates multiple queries using keyword search, terms and connectors, boolean, and Parallel Search to identify the on-point case law, statutes, and regulations, reads and analyzes the search results, and outputs a summary of its findings (i.e. an answer to the question), along with the supporting sources and applicable excerpts.

Prepare for a Deposition

Get a thorough deposition outline in no time, just by describing the deponent and what’s at issue.

How this skill works

  • Describe the deponent and what’s at issue in the case, and CoCounsel identifies multiple highly relevant topics to address in the deposition and drafts questions for each topic.

  • Refine topics by including specific areas of interest and get a thorough deposition outline.

Extract Contract Data

Ask questions of contracts that are analyzed in a line-by-line review

How this skill works

  • Allows the user to upload a set of contracts and a set of questions

  • This skill will provide an answer to those questions for each contract, or, if the question is not relevant to the contract, provide that information as well

  • Upload up to 10 contracts at once

  • Ask up to 10 questions of each contract

  • Relevant results will hyperlink to identified passages in the corresponding contract

Contract Policy Compliance

Get a list of all parts of a set of contracts that don’t comply with a set of policies.

How this skill works

  • Upload a set of contracts and then describe a policy or set of policies that the contracts should comply with, e.g. "contracts must contain a right to injunctive relief, not merely the right to seek injunctive relief."

  • CoCounsel will review your contracts and identify any contractual clauses relevant to the policy or policies you specified.

  • If there is any conflict between a contractual clause and a policy you described, CoCounsel will recommend a revised clause that complies with the relevant policy. It will also identify the risks presented by a clause that does not conform to the policy you described.

Summarize

Get an overview of any document in straightforward, everyday language.

How this skill works

  • Upload a document–e.g. a legal memorandum, judicial opinion, or contract.

  • CoCounsel will summarize the document using everyday terminology.

Search a Database

Find all instances of relevant information in a database of documents.

How this skill works

  • Select a database and describe what you're looking for in detail, such as templates and precedents to use as a starting point for drafting documents, or specific clauses and provisions you'd like to include in new documents you're working on.

  • CoCounsel identifies and delivers every instance of what you're searching for, citing sources in the database for each instance.

  • Behind the scenes, CoCounsel generates multiple queries using keyword search, terms and connectors, boolean, and Parallel Search to identifiy the on-point passages from every document in the database, reads and analyzes the search results, and outputs a summary of its findings (i.e. an answer to the question), citing applicable excerpts in specific documents.

Skills

UNIVERSAL
Search a Database

Find all instances of relevant information in a database of documents.

Summarize

Get an overview of any document in straightforward, everyday language.

Draft Correspondence

Rapidly draft common legal letters and emails.

TRANSACTIONAL
Contract Policy Compliance

Get a list of all parts of a set of contracts that don’t comply with a set of policies.

Extract Contract Data

Ask questions of contracts that are analyzed in a line-by-line review

Prepare for a Deposition

Get a thorough deposition outline by describing the deponent and what’s at issue.

LITIGATION
Legal Research Memo

Get answers to your research questions, with explanations and supporting sources.

Review Documents

Get comprehensive answers to your questions about a set of documents.