A 2024 academic study found hallucinations by major AI products in the legal market. The study should be read with caution in 2026.  While it recognizes value in the AI products, it reports flaws in what appear to me to be older versions of the products.

After a preprint version was posted, it was “subsequently peer-reviewed and published in the Journal of Empirical Legal Studies in 2025….”  Westlaw AI and Lexis+ AI Still Hallucinate: What the Stanford Study Actually Found – LegalAIWorld (undated); see Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, Daniel E. Ho, Hallucination‐Free? Assessing the Reliability of Leading AI Legal Research Tools – Magesh – 2025 – Journal of Empirical Legal Studies – Wiley Online Library (published Apr. 23, 2025).

The scholarly article concludes by emphasizing both the value of, and the need to verify, A.I. output.  Verification, of course, is not only good advice, but also an ethical mandate.  It is worth reviewing the study.

SUMMARY OF THE SCHOLARLY STUDY

Six researchers from Stanford and Yale posted a preprint version of “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools.”[1]

The Abstract states:  “While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time. We also document substantial differences between systems in responsiveness and accuracy.”

However, the article goes much further.  The authors state:

Commercially-available RAG-based[2] legal research tools still hallucinate. Over 1 in 6 of our queries caused Lexis+ AI and Ask Practical Law AI to respond with misleading or false information. And Westlaw hallucinated substantially more—one-third of its responses contained a hallucination.

On the positive side, these systems are less prone to hallucination than GPT-4, but users of these products must remain cautious about relying on their outputs.

The study’s authors provided specific examples in support of their conclusions.

EXAMPLES OF A.I. ERRORS FOUND BY THE RESEARCHERS

The study’s authors define “hallucination” as “a response that contains either incorrect information or a false assertion that a source supports a proposition,”[3] and “focus on factual hallucinations.”

Vendor Assertions

The article quotes vendor assertions as follows:

The following are official statements from Lexis, Casetext, and Thomson Reuters; however, none of them has provided any clear evidence so far to support their claims about the capabilities of their AI-based legal research tools:

Lexis: “Unlike other vendors, however, Lexis+ AI delivers 100% hallucination-free linked legal citations connected to source documents, grounding those responses in authoritative resources that can be relied upon with confidence.” (Wellen, 2024a) (emphasis added).

Casetext:[4] “Unlike even the most advanced LLMs, CoCounsel does not make up facts, or ‘hallucinate,’ because we’ve implemented controls to limit CoCounsel to answering from known, reliable data sources—such as our comprehensive, up-to-date database of case law, statutes, regulations, and codes—or not to answer at all.” (Casetext, 2023) (emphasis added).

Thomson Reuters: “We avoid [hallucinations] by relying on the trusted content within Westlaw and building in checks and balances that ensure our answers are grounded in good law.” (Thomson Reuters, 2023) (emphasis added). “We’ve all heard horror stories where generative AI just makes things up. That doesn’t work for the legal industry. They have to trust the content that AI serves up. With Ask Practical Law AI, all the responses are based on the expert resources of Practical Law.” (Thomson Reuters, 2024b) (emphasis added)

Examples Provided in the Article

The article states that the authors “document substantial variation in system performance. LexisNexis’s Lexis+ AI is the highest-performing system we test, answering 65% of our queries accurately. Westlaw’s AI-Assisted Research is accurate 42% of the time, but hallucinates nearly twice as often as the other legal tools we test. And Thomson Reuters’s Ask Practical Law AI provides incomplete answers (refusals or ungrounded responses; see Section 4.3) on more than 60% of our queries, the highest rate among the systems we tested.”

The authors add: “To offer one simple example…, the Westlaw system claims that a paragraph in the Federal Rules of Bankruptcy Procedure (FRBP) states that deadlines are jurisdictional. But no such paragraph exists, and the underlying claim is itself unlikely to be true in light of the Supreme Court’s holding in Kontrick v. Ryan, 540 U.S. 443, 447-48 & 448 n.3 (2004), which held that FRBP deadlines under a related provision were not jurisdictional.”

They describe one 2024 study in which Lexis AI+ reportedly cited a case decided in 2025.  They add:

For example, consider this Lexis+ AI exchange:

Prompt: What are some notable opinions written by Judge Luther A. Wilgarten?

Lexis+ AI Response: One of the notable opinions written by Judge Luther A. Wilgarten is the case of Luther v. Locke. The case citation is 2010 U.S. Dist. LEXIS 144868 and it was decided in the Eastern District of Virginia on January 11, 2010. In this case, the plaintiff had filed a Notice of Appeal of Judge Ellis’s decision, but failed to properly prosecute the appeal. […]

The article states:

While the retrieved citation offered is a real case and hence “hallucination-free” in a narrow sense, it was not written by Judge Wilgarten, a fictional judge who never served on the bench (Miner, 1989). And while the generated passages are based on the actual case, the second sentence contradicts the premise, suggesting Judge Ellis wrote the opinion, but the opinion was actually written by Judge Brinkema (and involved a prior decision by Judge Ellis, which forms the basis for the RAG response). Nor is the decision notable, as it was an unpublished opinion cited only once outside of its direct history. Hallucinations are compounded by poor retrieval and erroneous generation.

The article suggests that “Westlaw asserts that a U.S. Supreme Court case was reversed by the Nebraska Supreme Court on a matter of federal law. That is not possible in the U.S. legal system, and in fact the Nebraska Supreme Court did not so much as cite the Supreme Court case in question….”

The article continues:  “Lexis+ AI describes a rule established in Arturo D. as good law, with citation to the case that actually overrules Arturo D.”

It adds: “[W]hen asked to define the  ‘moral wrong doctrine,’ a doctrine pertaining to mistake-of-fact instructions in criminal prosecutions for morally wrongful acts…, Lexis+ AI relies on a source which defines moral turpitude, a legal term of art with a seemingly similar but actually unrelated meaning.”

SOME LIMITATIONS OF THE STUDY AND THIS BLOG POST

The paper was updated on May 30, 2024, which may be light-years ago in the world of artificial intelligence.  The tools studied were LexisNexis’s Lexis+ AI, Thomson Reuters’s Ask Practical Law AI, and Westlaw’s AI-Assisted Research.

There is no mention of West’s CoCounsel (released in August 2025) or Lexis’ Protégé (released in January 2025).  The authors expressly note that the programs they studied are “emerging systems.”  They point out that “our evaluation only captures a point in time. Even over the course of our study, we noticed the responses of these systems—particularly Lexis+ AI—evolve over time.”  And, the authors wrote:

Since the completion of our evaluation for this paper in April 2024, LexisNexis has released a “second generation” version of its tool. Our results do not speak to the performance of this second generation product, if different. Accompanying this release, LexisNexis noted, “our promise is not perfection, but that all linked legal citations are hallucination-free” (LexisNexis, 2024).

The study candidly notes: “[O]ur primary goal is limited to assessing the hallucination rate, accuracy, and groundedness on emerging legal technology. These are central concepts to the trustworthiness of AI tools, but they are not the sole criteria for the quality and value of a legal research system. For instance, notwithstanding the many hidden hallucinations, the overall output of Lexis+ AI and AI-AR may still be quite valuable for distinct use cases (e.g., starting on a research thread).” (emphasis added).

The authors describe the study as “the first systematic assessment of leading AI tools for real-world legal research tasks.”  This blog post is a lay-reader’s summary of a complex scientific study.  The study manually contruct[ed] a preregistered dataset of over 200 legal queries….” The technological analysis is far beyond my capabilities, and the authors made their dataset, tool outputs, and labels available to others more qualified than I.  As such, I make no effort to delve into the methodology, reliability, or validity of the study.

And, in this post, I do not address the stated limitations of the article.^[5]  For example, and without limitation, the authors state that they do not use the “gold-standard” on some issues. See n. 8. However, methods such as “[w]ith this protocol, we find a Cohen’s kappa (Cohen, 1960) of 0.77 and an inter-rater agreement of 85.4% on the final outcome label (correct, incomplete, or hallucinated) between the evaluation labeler and the initial labels,” are far beyond my skill-set.

THE ARTICLE CONCLUDES THAT THESE A.I. TOOLS PROVIDE VALUE

Importantly, the researchers state: “[E]ven in their current form, these products can offer considerable value to legal researchers compared to traditional keyword search methods or general-purpose AI systems, particularly when used as the first step of legal research rather than the last word.”

The paper concludes: “AI tools for legal research have not eliminated hallucinations. Users of these tools must continue to verify that key propositions are accurately supported by citations.”  That is indisputable.

THE STUDY’S RESULTS HAVE BEEN QUESTIONED

“The study was not received quietly.”  Westlaw AI and Lexis+ AI Still Hallucinate: What the Stanford Study Actually Found – LegalAIWorld  Both Thomsen Reuters and Lexis disputed the findings.

For example: “Thomson Reuters said that their internal testing showed a lower hallucination rate compared to the study, and welcomed the opportunity to work with Stanford to explore creating AI benchmarks.” Isha Marathe, Updated Stanford Report Finds High Hallucination Rates on Westlaw AI | Law.com (Jun. 4, 2024)(behind pay wall).

One article states that:

Both companies’ objections deserve to be taken seriously. This is not a clean, uncontested piece of research. The methodology was imperfect in its initial form, the access restrictions created real limitations, and the vendors’ own systems have been updated since the study was conducted. Any fair account of this study has to include those caveats.

Westlaw AI and Lexis+ AI Still Hallucinate: What the Stanford Study Actually Found – LegalAIWorld

SOME OF THE QUESTIONS HAVE BEEN QUESTIONED

However, the Legal AI World blog adds:

And the vendors’ response to that finding was not to publish their own independent benchmarks proving otherwise. It was to dispute the methodology and point to internal data they have not made public.

In the absence of transparent, third-party benchmarking — which the Stanford researchers explicitly called for — lawyers are being asked to trust marketing claims that have not been independently verified. That is a professional responsibility problem, not just a product quality question.

Id.  “As the Stanford researchers argued, what the legal profession needs is public benchmarking of these tools — conducted independently, using preregistered methodology, updated regularly as the products improve.”  Id.  Legal AI World concludes:

The hallucination problem in legal AI has not been solved. Not by LexisNexis. Not by Thomson Reuters. Not by any legal AI product currently on the market. The research is unambiguous on this point, and every lawyer using these tools needs to proceed accordingly.

Id.

POSTSCRIPT

So, in addition to the passage of time and my lack of scientific or technological qualifications to review the methodology, my major caveat includes the fact that this post merely skims the surface of the manuscript.  The study has garnered a lot of attention. E.g., Bob Ambrogi, In Redo of Its Study, Stanford Finds Westlaw’s AI Hallucinates At Double the Rate of LexisNexis | LawSites (Jun. 2, 2024); AI Legal Tools caught hallucinating Again – Stanford Study; Westlaw AI and Lexis+ AI Still Hallucinate: What the Stanford Study Actually Found – LegalAIWorld.  It is an important piece of scholarship.

There are so many “hallucination” cases that they cannot reasonably be listed here.  In Farris, the Sixth Circuit wrote:

New technologies, moreover, are no substitute for tried-and-true safeguards managed by practicing attorneys. Attorneys have an ethical obligation to verify the citations and propositions they submit to courts; that obligation reflects duties of competence and candor that apply no matter the tools attorneys use.… So, attorneys who rely on artificial intelligence must remain diligent in supervising their work product and carefully examine the accuracy of every citation they present to this Court. Here, Howe’s reliance on “staff”—rather than himself or another attorney—to supervise the artificial-intelligence-generated work product fell short of his obligations as attorney of record. See Model Rules of Pro. Conduct. r. 5.3 (A.B.A. 2012); Ky. Sup. Ct. R. 3.130(5.3) (2022).

That Howe’s briefs cited real legal authorities—as opposed to “hallucinations” featuring fictitious cases—does not absolve him. See Sanders v. United States, 176 Fed. Cl. 163, 169 & n.8 (2025) (collecting cases with invented authorities). Howe’s failure to verify the artificial-intelligence output still resulted in the submission of false quotations and misleading legal arguments to this Court. Again, attorneys’ professional duties demand more.

Id. at *3.  While the academic article does not appear to be applicable to current West, Lexis, and other models, and even though I cannot evaluate or comment on the study’s methodology, the paper is valuable for demonstrating the need to validate all AI output in the context of RAG tools for the legal industry.

I will post any reasonable response or comment from West, Thomsen Reuters, Lexis, or any person or entity cited in this blog.

_____

[1] See also AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries | Stanford HAI (May 23, 2024).

[2] “RAG” stands for retrieval-augmented generation.  RAG tools reduce hallucination by confining the retrieval to specific data.

[3] But see §4.3 of the paper for additional precision.

[4] Thomson Reuters Completes Acquisition of Casetext, Inc. – Thomson Reuters Institute (Aug. 17, 2023).

[5] See §7.

He posits a “fundamental principle”—“ask for the original,” writing that doing so is the “single most important discovery strategy for authenticating digital evidence….”  He then suggests asking for both application and file system metadata and provides sample discovery requests.  To give only one of many examples, he suggests a request for admission that the proponent: “Admit that the photograph marked as Exhibit A was created by a digital camera or smartphone, not by artificial intelligence software.”

Craig discusses how metadata “has become the last line of defense against manufactured reality.”  He wrote that “critically, when digital evidence is fabricated, this metadata is almost always absent, incomplete, or inconsistent.”  He added:

Here’s the critical insight: AI systems that generate synthetic images don’t create authentic metadata. They aren’t cameras. They’re software programs running on computers…..

Like all of his publications, Craig’s work provides practical tips.  He discusses, for example, “several telltale metadata anomalies….”  One suggestion is a “contradiction search”:

What evidence might contradict the offered evidence?  If it purportedly shows the plaintiff at a particular location, do cell phone records, credit card receipts, or testimony place them elsewhere?

Craig’s thesis is that “[m]etadata can’t be authentically fabricated because authentic metadata requires an authentic source.”  Thus, he suggests: “Ask the foundational questions: Where did this come from?  What device created it?  What does the metadata show? What should be here that isn’t?”  While Craig states that metadata is not foolproof—“sophisticated fabricators may attempt to spoof provenance, and legitimate evidence may lose its metadata through innocent handling”—he concludes that “metadata and forensic rigor are out anchors to truth.”

Doug Austin’s review is posted at A Practitioner’s Guide to Detecting Deep Fakes and Authenticating Digital Evidence (2026).

See also Book Review: Craig Ball, “The Leery Lawyer’s guide to AI and LLMs in Trial Practice” – E-Discovery LLC (Jan. 16, 2026); Book Review: Tom O’Connor, “Artificial Intelligence for the Rest of Us” – E-Discovery LLC (Jan. 12, 2026).

For additional reviews of recent Artificial Intelligence books, please click on the “Book Reviews” “Category” above.

For additional posts on A.I., please click on the “artificial intelligence” “Tag” above.

For another discussion of approaches to deepfakes, see Deepfakes – National Center for State Courts  (Mar. 6, 2026).

Craig is a national resource.  A list of his contributions to this field would fill a book. Craig ran the prestigious Georgetown E-Discovery Training Academy, a week-long intensive ESI program. He has served as a special master, a consultant in computer forensics, forensic expert, law school professor, lecturer, writer, and trial lawyer.  We have worked on publications together, taught together at the Judicial College of Maryland, served on the ESI Principles subcommittee for the U.S. District Court for the District of Maryland together, and otherwise had many good times, such as his guest lectures to my University of Baltimore Law School ESI class.

Craig’s latest contribution is “The Leery Lawyer’s Guide to AI.”  In her blog Review: How “The Leery Lawyer’s Guide to AI and LLMs in Trial Practice” Made Me a More Confident Legal Tech User – EDRM (Jan. 13, 2026), Sheila Grela  wrote:

Many AI articles are steeped in jargon and abstract promises. So when I came across Craig Ball’s The Leery Lawyer’s Guide to AI and LLMs in Trial Practice, I found something different and extremely useful. This is not theory dressed up in tech-speak; it’s a practical manual for professionals navigating real legal work…. One thing that stood out immediately was how grounded the guide is. It skips the usual tech jargon and focuses on what matters: helping lawyers and their teams work more efficiently without compromising quality.

I agree.

In 2026 Guide to AI and LLMs in Trial Practice by Craig Ball! (Jan. 14, 2026), Doug Austin wrote “the guide is all practical, as in practical applications of the models for litigation and trial practice, followed by ten examples with prompts, ten tips for improved prompts, guidance for understanding limitations and ethical guardrails, and final thoughts for trial lawyers. Craig finishes the guide with an Appendix on an AI prompt to critique and improve keyword searches (an adaptation of one of his past blog posts).”

I have little to add.  But, I’ll try.

Craig wrote: “Used prudently, [AI] functions as the world’s fastest junior associate—minus the hourly rate and parking validation.”  For example:  “One of the simplest and most effective ways to improve your prompts is by asking an LLM to do it for you.  I’ll often give ChatGPT a prompt and then ask it to streamline and improve my prompt before I run it.”  He adds that attorneys must “remain firmly in the driver’s seat.”

I can’t summarize all of Craig’s prompting guidance. But there are many very interesting suggestions.  For examples of prompt design:

• “Use only the materials I provide; do not rely on outside knowledge.”
• “If the record is silent or ambiguous, say so.”
• “End with a list of assertions that require human verification.”

Craig explains key terms, such as “tokens” and “context windows.”  He points out upload issues with large files, how to “chunk” uploads, and he discusses when “older material is displaced” in long sessions.

Two of my favorite use case suggestions are:

“Search Term Generation and Query Assessment

Turning plain-English descriptions into search terms or Boolean queries for discovery is another area where LLMs perform well. They can analyze relevant documents and propose keyword lists, proximity connectors, and alternative formulations based on how people actually communicate rather than how lawyers guess they might. LLMs can also critique existing search terms, identifying overbreadth, redundancy, or likely blind spots. This is useful both for improving your own searches and for evaluating the adequacy of an opponent’s proposed terms.”

“Meet-and-Confer and Proportionality Advocacy

AI tools can assist in drafting meet-and-confer correspondence that explains discovery positions in plain English: why certain custodians matter, why others do not, and how proposed limits reflect proportionality. This is especially useful when translating technical search methodology into language suitable for opposing counsel—or the court. Used carefully, LLMs help sharpen the explanation of discovery positions, not the positions themselves. They are best employed to clarify rationale, not to invent it.”

In that regard, Craig provides the following:

Prompt Example: “Draft a meet-and-confer letter proposing a proportional ESI protocol for a federal employment discrimination case in Texas. Address data sources, custodians, date ranges, search methodology (with examples), metadata fields, production format, and privilege clawback. Explain the rationale for each choice in plain English.”

Tip: you might try uploading my free primer on ESI Protocols along with your prompt:  http://www.craigball.com/ESIProtocol.pdf

However, these limited examples are only a very small sampling of use cases that Craig covers.

Craig cautions: “Finally, remember that LLMs reason statistically, not legally. They recognize patterns in language, not truth. When the record is thin, ambiguous, or contested, the risk of confident but wrong output increases. The antidote is disciplined prompting, scoped inputs, and explicit verification requirements—habits already familiar to good trial lawyers.”  His closing sentence is: “Just remember: every tool needs a craftsman, and no amount of technology can replace the judgment and insight of a seasoned trial lawyer.”

This the fourth in a series of reviews of books on artificial intelligence. Click on the “Artificial Intelligence” TAG, above, for other reviews.

I had previously read John C. Tredennick, et al.,  TAR for Smart People – Google Books (Catalyst 2019), and was impressed.  The recent Generative AI book did not disappoint.

Mr. Tredennick tells the reader what he is going to explain, explains it, and then reminds the reader what has been explained. The writing is clear and concise. For example: “LLM’s are ‘brains in jars’ without memory, whose only interface with the world is a temporary ‘whiteboard’ onto which users write their instructions.”

Important concepts, such as “pre-training” and “training cutoff”  are covered.  One very interesting topic is that of “context windows” and how their size is significant.  Mr. Tredennick explains that the “context window” is the “whiteboard” for the “brain in a jar.”  The book also covers the confidence level of AI systems, which is a “numerical measure of how certain an AI system is about its prediction or classification.”

The topic of “hallucinations” is explored.  More importantly, the book describes lower-risk scenarios and mitigation strategies, as well as emphasizing the need for verification.  One example is that summarization or extraction presents a lesser risk than generation. Another is a discussion of Retrieval Augmented Generation or “RAG.”

A significant discussion is that of “verification strategies,” including sampling, risk-based verification, cross-checking, and others.  Of course, the “human-in-the-loop” “imperative” is emphasized.

Ethical issues are covered.  In addition to the ABA Formal Opinion, there is a discussion of “transparency and client communication,” as well as supervisory and billing considerations.

Mr. Tredennick also gives valuable step-by-step examples of sequential queries, privilege analysis, analyzing contradictory testimony, timeline construction, and evaluating opposing arguments. For example, he wrote that AI can “systematically compare testimony” and find “what different witnesses said about the same events.”  Further, he suggests that “timeline systems” perform “gap analysis” and “identify not just what’s present but what’s conspicuously absent.”

Part III covers “practical applications in discovery and investigations.” In addition to responsiveness and privilege reviews, it discusses use of AI to find passages meeting redaction criteria, as well as transcript and medical record analysis.  Mr. Tredennick points to a real-world scenario in which 102 transcripts, totaling 18,000 pages, were summarized in 131 minutes, providing complete summaries, structured outlines, topic tables, and page-line hyperlinks to the original text.

The book discusses zero-shot, one-shot and few-shot prompts. This is a prompt engineering process that involves whether and how to provide examples to the AI.  Tredennick provides the “anatomy of an effective prompt.”

One last (for this blog, but not the book) important point, is that Mr. Tredennick states: “Repository quality determines output quality.”  He notes that AI “can only work with the materials it is given.” If the repository in incomplete, the results will be less than optimal.

This blog was initially posted on  Electronic Discovery Reference Model.

The thesis of the book is that “computers are capable of reviewing and classifying document better than humans.  And that’s a big deal in eDiscovery.” As its title suggests, the book is focused on AI document review, and contrasting it with TAR and predictive coding. While Technology Assisted Review uses humans to train the machine, AI is trained and uses prompts to tell it what to look for. It does not use “training examples.” Mr. Sullivan provides a sample instruction:

“All documents where an Acme employee suggests that pricing of widgets should be modified.”

You’ll notice the instructions read like a Request for Production, which is exactly what they are. In most cases, we simply copy the exact language from the Request for Production to start our instructions

Mr. Sullivan writes that “AI-powered review… can easily find 95%+ of the relevant documents.” I thought the “how to” chapters were among the most interesting. The book walks through a relevancy review, step-by-step, using random sampling to “QC,” or quality control, the results.

As a validation process, Mr. Sullivan follows the tried and true path of classifying true positives, true negatives, false positives, and false negatives, to create metrics such as recall and precision. In my experience, these techniques have long been used on, for example, keyword searches.  Here, they are applied to AI.  The book provides the simple formulae:

Recall = TP/(TP + FN)

Precision = TP/(TP + FP)

The author describes preparation of an “answer key” by a subject-matter expert.  He uses a term that I have not heard to describe the process of calculating metrics with that key – a “confusion matrix.”  The process applies standard techniques, such as sampling the “discard pile,” to improve iterative queries. Frankly, this blog shortchanges Mr. Sullivan’s excellent chapters because it would take too long to summarize them.

As to defensibility, Mr. Sullivan wrote: “The only thing that matters is how you validate the results and demonstrate high-quality output.”  While (in my opinion) validation may not be the “only” thing, its importance cannot be overstated. The author explains:

So, what does a defensible AI Review look like? It’s a lot like any Predictive Coding review. We need to use sampling to validate the results. Let’s walk through how we can do that. The general process for predictive coding has become pretty straightforward:

1. Identify the review set.
2. Train the machine.
3. Run the documents through the classifier.
4. Evaluate the results.

Believe it or not, it’s no different with AI.

The book is full of concrete examples.  For step 1, for example, it suggests removal of ROT (redundant, obsolete, or trivial), documents without extracted text, audio files, images, and huge files, as well as deduplication. That is the classic approach to document review.

Mr. Sullivan also suggests “pre-validation.”  This consists of running prompts against a random sample before running them against the full data set.  Then a subject-matter-expert reviews the “hits” to determine recall and precision. This provides a benchmark that is analogous to what I have called “richness.”  Mr. Sullivan suggests pre-validation as a cost-saving measure.

Another excellent discussion is that prompts may be refined by either inclusion or exclusion criteria.  An example of inclusion criteria is that “any discussion about qualifications in hiring should be deemed relevant.” Exclusion would be: “Any discussion about hiring anyone other than coaches or management should be considered not relevant.”

Mr. Sullivan discusses full AI review, but also posits options such as “AI-Powered Linear Review,” in which batches are selected using AI, and AI/CAL Hybrid Review, in which seed documents are reviewed by AI.

As to confidentiality and security, Mr. Sullivan wrote: “If you aren’t paying for a product, you are the product.” He offers questions to ask the AI provider to ensure security.

The book ends with: “If you have any questions or comments, or you want to talk over any strategies you are considering, we can always be reached at support@ediscoveryai.com.

This blog was initially posted on  Electronic Discovery Reference Model.

I reviewed Tom’s prior book, “Ediscovery for the Rest of Us,” in Book Review:  Any Ship Can Be a Minesweeper – – –  Once (Apr. 28, 2023). In that blog, I said that, when it comes to electronically stored information, Tom O’Connor is a national treasure.  He has a wealth of litigation-related ESI experience, helped run the prestigious Georgetown E-Discovery Academy, lectures on recent cases and developments in the law, has published several books and blogs, and is a consultant.  He has generously shared his encyclopedic knowledge with countless less-experienced attorneys and other legal professionals. I added:

There is an old military saying that “any ship can be a minesweeper – – – once.”  Tom’s latest book shows general practitioners and business executives how to avoid the mines. For example, he discusses: ESI budgeting; do-it-yourself options; use of native files and phasing to reduce cost; and when a vendor or consultant is needed, and when one is not necessary.

His newest book on AI “for the rest of us” follows in that tradition. Tom counsels, in the words of the NY State Bar Association, “be cautious, be curious, be vigilant, be brave.” That reminds me of a phrase anonymously attributed to a U.S. Marine Gunnery Sargeant: “You, you, and you… Panic. The rest of you, come with me.”

Like ABA Formal Op. 512, Tom views AI as “a springboard or foundation for legal work….”  It is a means to an end. Tom calls it an “enhancement tool.”

Still, at the end of the day, AI is really just a stochastic[1] parrot.  It finds statistical relationships in massive data sets to convincingly generate human-like text, but lacks the true semantic understanding behind the word patterns.

Tom quotes Miles Kington: “Knowledge is knowing that a tomato is a fruit; wisdom in not putting it in a fruit salad.”  AI can help spot patterns.  But, Tom correctly argues: “The real intelligence? That still comes from lawyers.”

The Introduction, by Mr. Madhava, makes the point that automating and speeding up tasks is exciting and “prudence does not require paralysis.”  The book begins with a description of AI and states that “the ‘reasoning’ to which AI developers refer differs significantly from human reasoning….”

Tom provides a brief history of AI. Listing twelve legal use cases, Tom suggests asking two critical questions: “First, ‘What specific problem is this tool trying to solve?’ Second, ‘What type of AI technology is it employing to address that challenge?’”  Obviously, it would be ill-advised to deploy a contract generation tool for use in e-discovery and evidence review.  In that regard, the book lists twelve questions to pose to an AI vendor.

In an interesting experiment, Tom asked four different GenAI models to identify the Top 5 use cases for AI in the legal field. The results were not consistent. For example:

• Three of the four models topped the list as “legal research.” However, another listed “legal document review and analysis” as the top use case.
• Two models ranked “contract review” second. One ranked legal research, and another listed “document review and analysis,” in that category.

One valuable discussion is Tom’s seven practical tips for implementing AI.  They include, as a starting point, defining the problem before selecting technology. And, Tom points to a quality control issue that I have experienced: “You don’t want to spend more time checking AI outputs than you would have spent doing the work yourself.”

Tom cites a 2025 survey by eDiscovery Today that found that only 12.2% of the respondents are not using GenAI at all.  He points to most of the usage on the “right side” of the EDRM for summarizing documents, answering questions directed against a body of evidence, and looking for contradictions.  He suggests that operations on the “left side” of the EDRM, such as early case assessment, have received slower acceptance.

The book suggests including AI in an ESI Protocol.[2]  It suggests discussing how the technology may (or may not?) be used. That is an important and hotly-debated topic.  See “Against an AI Privilege” – Are Prompts Discoverable?  Is Output? (Jan. 2, 2026).

The ethics discussion is comprehensive. In very brief summary, it lists several ethical duties: 1) competence by understanding the technology; 2) confidentiality by protecting client information; 3) supervision of the use of AI; and, 4) recommended client disclosure.

The Glossary provides useful definitions of critical terms, such as “tokens,” “recall,” “precision,” “parameters,” “Large Language Model,” and “Retrieval-Augmented Generation” (“RAG”).  It even points out that the “GPT” in ChatGPT stands for “Generative Pre-Trained Transformer.”

This blog was initially posted on  Electronic Discovery Reference Model.

____

[1] “A stochastic process or system is connected with random probability.” STOCHASTIC | English meaning – Cambridge Dictionary

[2] An “ESI Protocol” is Not a Rule 26(f) “Discovery Plan” (Nov. 24, 2025).

Entertaining:  Watergate Prosecutor Archibald Cox was “a man of grace and dignity….”  Mr. Honig reports that Mr. Cox “once noticed an establishment near the office on 14^th and K Streets in Washington, D.C….” It was called “Archibald’s.”  And, “Cox suggested that [he and his staff] might go eat there sometimes, given that he shared a name with it.”  Mr. Honig reports, however, that younger prosecutors “knew that Archibald’s was in fact a strip club” and they told Cox, “[t]hat place isn’t for you, Archie.”

Another anecdote of interest is:

“Irresistible fun fact: In 1864, [President Abraham] Lincoln pardoned Moses J. Robinette, a civilian in the Union Army who had stabbed another Union employee during a fight in a mess tent and was sentenced to two years of hard labor. Robinette was the great-great-grandfather of the future president, Joe Biden.

Educational:  Although subtitled “from Nixon to Trump,” the book begins earlier, with a historical description of Pres. Ulysses Grant’s appointment of, and efforts to later derail, an “outside prosecutor.”  In fact, Mr. Honig covers much more, from Grant to Truman, as well as Iran-Contra under President Reagan, Ken Starr and President Clinton, and the investigation of President Biden.

The book summarizes a broad presidential strategy of bringing in an outside investigator to give an appearance of independence, only to later restrict it.  In early days, there were no statutes and this resulted in ad hoc approaches with the President retaining “virtually unrestrained power over the unfolding cases…”  Thus, “both Grant and Truman removed the attorney general or outside prosecutor, without endangering their own presidencies,” according to Mr. Honig.

The book explains the change from a “Special Prosecutor” to an “Independent Counsel,” to “Special Counsel,” and provides an in depth look at the Watergate prosecution, among others.  It states: “Total costs of the investigations ran from less than $20,000 to more than $73 million.  Some investigations lasted fewer than six months, and others took more than ten years.”

However, Mr. Honig suggests that the “playbook” often went back to President Grant – – appoint an outside counsel to fend off political pressure, and then throw up roadblocks if the investigation becomes uncomfortable.  Mr. Honig wrote: “Targets of investigations attack the motives of the prosecutors…. And terms like ‘witch hunt’ and ‘fake news’ were coined by investigative targets well before Donald Trump came along…. The point is, as we move forward: Everything old is new again.”

Disturbing:  The book is both comforting – real investigations were conducted and led to convictions – and disturbing – very real flaws existed.  It explains failures by prosecutors, defects in statutes and regulations, and, questionable judgment calls, as well as unacceptable political involvements.

Further, it describes in detail how the scope of presidential immunity from criminal prosecution remains very much unresolved.  Mr. Honig posits a hypothetical.  Would the following hypothetical text from a sitting president to the attorney general be immune?

Opposition party leader in the Senate is giving me a headache on this bill. Need you to arrest him for whatever you can think of – maybe drugs or child pornography?  Just plant something at his house, do a search warrant (make up whatever you need to get the judge to sign, pick a friendly one). Notify press in advance, have them standing by when you go in.  Get this done and the next Supreme Court opening is yours.

Mr. Honig explains that the answer is unclear. The fact that it is even debatable is unsettling.

Prescriptive & Optimistic:  Mr. Honig provides an analytical framework.  He posits that each historical precedent can be evaluated using six criteria.

1. Necessity:  Did we need an outside investigative attorney?  Could DOJ have handled it in the ordinary course of business?
2. Duration: Was the investigation timely concluded or did it drag out?
3. Scope: Was the scope clearly defined or was it infected by “mission creep”? Was it “focused” or a “wild-goose chase”?
4. Convictions: Did the investigation result in criminal charges or convictions?
5. Declinations: This is the “inverse of the prior question,” according to Mr. Honig. “Sometimes a prosecutor’s job is not to prosecute if the facts don’t conclusively establish criminality.”
6. Sixth: How was the investigation perceived by the public? Was it credible?

The need for evaluation and, to use Mr. Honig’s term, “guardrails,” is self-evident. The book reports an anonymous comment to Independent Counsel Donald Smaltz in 1988: “G-d knows, if I had $30 million, I could find dirt on you, sir.”

With those six factors in mind, the last chapter is titled “The Next Evolution.”  It suggests new “guardrails” that are practical, efficient, and constitutional.  Mr. Honig lays out a seven-point plan and concludes: “There’s no perfect solution. But in the final analysis, much as Winston Churchill famously said of democracy itself: It’s the worst form of government, except for all others.”  That is cause for optimism.  To paraphrase an old saying, “from your lips, to Congress’s ears.”

My comment is that the Commentary is excellent and well worth reading.

I am not going to try to summarize the comprehensive, 32-page document.  Instead, I will discuss one issue, “document unitization.”

The Commentary defines “document unitization,” as “dividing continuous communication strings such as chat or instant messages into more manageable units, such as a 24-hour period….”

Sedona suggests unitization methodologies, such as “before and after” agreements and use of date ranges, as well as their advantages and disadvantages.

As to date range unitization:

For chat and instant-message data, consider whether the collection should be limited to certain date ranges. Unitizing (i.e., breaking up) continuous message strings into smaller pieces is becoming a common practice. Options include breaking up strings into certain time periods (e.g., 24 hours) or into a certain number of messages. If the relevant participants are all in the same time zone, it may be best to process in that local time zone and unitize at, for example, midnight. If the participants are in different time zones, then processing by UTC zone and unitizing either by 24-hour periods or by breaks in discussion may be preferred. The potential downside of unitization by time period or number of messages is that it can artificially break up continuing conversations into separate documents, requiring the parties to manually reassociate them for use in the litigation. [emphasis added].

As to “before and after” unitization agreements:

One method to organize responsive messages is to produce a certain number of messages before and after a responsive chat message. Another method is to identify and organize communications from a particular time range or date range. These are both forms of “unitization” in production. The potential downsides to these approaches include that there may be messages outside the selected message range that would also provide helpful context, and those messages would not be produced as a single conversation if the parties are following such a protocol. [emphasis added].

After noting the advantages and disadvantages of unitization, Sedona discusses the alternative:

Not breaking up the message string into different units, however, would likely result in both irrelevant and relevant messages being produced together. The production of such irrelevant messages may increase the risk that personal and sensitive information is produced. Communication within chat and instant-messaging applications may also jump from one subject to another in quick succession, and then back to earlier subjects. Breaking up the conversation into different units may therefore require the parties to reassociate the related messages manually….

The Commentary summarizes the current decisional authority as follows:

Some courts’ decisions reflect that information contained in continuous message streams may need context, and therefore even messages that by themselves may be considered irrelevant may nevertheless be discoverable because they provide needed context to other relevant communications. Other courts, however, have held that the producing party can unilaterally withhold portions of a text message chain that are not relevant to the case.

The unitization or context issue is not unique to collaboration platforms.

I have written a series of blogs on the topic of “what is a document?” “Modern Attachments” or “Pointers”- What is a Document? (Part IV) (Aug. 12, 2022).

In those blogs,[1] I pointed to parallel issues that may be presented by, for example, text bubbles, spreadsheet cells, Excel workbooks with multiple worksheets, and PDF Portfolios.  In each instance, one may reasonably ask “what is the document?”

Trial, or at least trial preparation, is the primary endgame of discovery:

The fundamental objective of discovery is to advance the sound and expeditious administration of justice by eliminating, as far as possible, the necessity of any party to litigation going to trial in a confused or muddled state of mind, concerning the facts that gave rise to the litigation.

Rodriguez v. Clarke, 400 Md. 39, 57, 926 A.2d 736, 747 (2007)(cleaned up; emphasis added).[2]

While Fed.R.Civ.P. 26(b)(1) states that “[i]nformation within this scope of discovery need not be admissible in evidence to be discoverable,” discoverable material that does not meet evidentiary standards may be of little or limited value.

In the words of the Hon. Paul W. Grimm (in a different context), “considering the significant costs associated with discovery of ESI, it makes little sense to go to all the bother and expense to get electronic information only to have it excluded from evidence or rejected from consideration during summary judgment because the proponent cannot lay a sufficient foundation to get it admitted.”  Lorraine v. Markel Amer. Ins. Co., 241 F.R.D. 534, 538 (D. Md. 2007).  Judge Grimm wrote that “[w]henever ESI is offered as evidence,… evidence rules must be considered….” [emphasis added].

Because discovery is not an end in itself, I have discussed, in detail, evidentiary issues that may arise when a “document” is truncated during the unitization and discovery process.  What is a “Document?” (Aug. 17, 2021).  And, of course, the need for “context” was addressed in decisions such as Sandoz v. Un. Therapeutics Corp., 2021 WL 2453142 (D.N.J. Jun. 16, 2021)(text bubbles).

Since the earliest days of e-discovery, parties have been advised to discuss the form or forms of production.  The Sedona Commentary emphasizes this – – and much more – – in the context of collaboration platforms.

In my opinion, one factor that should be considered in these discussions is how the produced communications can be used in depositions, motions, and at trial.  Fed.R.Evid. 106 states:

If a party introduces all or part of a statement, an adverse party may require the introduction, at that time, of any other part–or any other statement–that in fairness ought to be considered at the same time. The adverse party may do so over a hearsay objection.

The Advisory Committee Note provides an example: “[A]ssume the defendant in a murder case admits that he owned the murder weapon, but also simultaneously states that he sold it months before the murder. In this circumstance, admitting only the statement of ownership creates a misimpression….”  A similar issue may be presented in complex electronic communications.  For a more detailed scenario of the evidentiary issue, please see What is a “Document?”

____

[1] A single file may contain multiple documents.  What is a Document? (Part III) (Apr. 4, 2022)(Excel workbooks and Adobe Portfolios); What is a Document? (Part II) (Aug. 28, 2021)(user created data such as Excel formulas).

[2] “Civil discovery is a device to allow parties to obtain information for the purpose of preparing and trying a lawsuit.”  Gillard v. Boulder Valley Sch. Dist. Re.-2, 196 F.R.D. 382, 387 (D. Colo. 2000); Seattle Times Co v. Rhinehart, 467 U.S. 20, 34 (1984) (“Liberal discovery is provided for the sole purpose of assisting in the preparation and trial, or the settlement, of litigated disputes”); Hickman v. Taylor, 329 U.S. 495, 507 (1947)(“Mutual knowledge of all the relevant facts gathered by both parties is essential to proper litigation.”).

The publication illustrates the old military maxim that “any ship can be a minesweeper – once.”

The authors are Hon. Herbert B. Dixon Jr., Hon. Allison H. Goddard, Maura R. Grossman, Hon. Xavier Rodriguez, Hon. Scott U. Schlegel & Hon. Samuel A. Thumma.  They wrote:

These Guidelines are intended to provide general, non-technical advice about the use of artificial intelligence (“AI”) and generative artificial intelligence (“GenAI”) by judicial officers and those with whom they work in state and federal courts in the United States.

The Guidelines point out that “when judicial officers obtain information, analysis, or advice from AI or GenAI tools, they risk relying on extrajudicial information and influences that the parties have not had an opportunity to address or rebut.”

“Judicial officers and those with whom they work should be aware that GenAI tools do not generate responses like traditional search engines. GenAI tools generate content using complex algorithms, based on the prompt they receive and the data on which the GenAI tool was trained.” Id. at 4.

Additionally:

Users must exercise vigilance to avoid becoming “anchored” to the AI’s response, sometimes called “automation bias,” where humans trust AI responses as correct without validating the results. Similarly, users of AI need to account for confirmation bias, where a human accepts the AI results because they appear to be consistent with the beliefs and opinions the user already has.

The Guidelines warn that “there may be good reason to retain, or to disable or delete, the prompt history after each session.”

They also caution against potential AI bias.

The authors suggest that some use cases are for: legal research; drafting routine administrative orders; searchimg and summarizing depositions, briefs, and exhibits; creating timelines; and for editing or proofreading.  Id. at 6. The Guidelines provide a number of other possibilities. Id. at 7.

However:  “As of February 2025, no known GenAI tools have fully resolved the hallucination problem, i.e., the tendency to generate plausible-sounding but false or inaccurate information.”  Id. at 7.

EDRM has published a Repository of Judicial Standing Orders Including AI Segments.

See also A Review of Sedona’s “Artificial Intelligence (AI) and the Practice of Law” by The Hon. Xavier Rodriguez (Sep. 27, 2023).

EDRM has posted Hon. Ralph Artigliere (ret.), Ethical AI Guideposts for Lawyers Using Generative AI – EDRM (Oct. 31, 2023).

Phil’s publication is an annual event.  Book Review:  Phil Favro’s “Selected eDiscovery and ESI Case Law from 2022-23″;  Sedona Conference “Selected eDiscovery and ESI Case Law from 2021-22.”

As in the prior works, Phil’s new publication runs the gamut of topics and hits the important ones.  Each case summary is clear and to-the-point.  Summaries are organized by topic, such as cooperation, ESI Protocols, form of production, Fed.R.Evid. 502, “possession, custody, and control,” relevance redactions, sanctions, and many others.

I learn something new every time I read one of Phil’s blogs or publications.

Phil’s writings are posted on his company blog page,  Philip Favro – Innovative Driven.  Phil’s Sedona Conference biography states that he “is a leading expert on issues relating to the discovery of electronically stored information. Phil serves as a court-appointed special master, expert witness, and trusted advisor to law firms and organizations on matters involving electronic discovery and ESI. He is a nationally recognized scholar on electronic discovery, with courts and academic journals citing his articles. Phil also regularly provides training to judges on electronic discovery and ESI….”  Accord Philip Favro | Practical Law The Journal | Reuters.

He is also a very nice person.

It was an honor to be listed as one of several people who contributed.  As always, this is an excellent resource in an always-changing field.