The case is a putative collective and class action alleging a “nationwide pattern, practice, and/or policy of discriminating against older workers … in their hiring processes.”  At issue was the manner of collective notice procedures.

The parties raise four primary disputes regarding the proposed collective notice procedures: (1) whether notice should be authorized for applicants of the non-RTX Defendants, (2) how to determine which job postings qualify as “Recent Graduate Positions,” (3) how to identify putative collective members, and (4) whether a website to return consent forms should be publicly available or password protected.

Id. at *2.

This post focuses on No. 2. The court wrote:

Next, the parties dispute how to determine which job postings qualify as “Recent Graduate Positions” reserved for “those who have graduated in the prior year or two (or who will soon graduate), or applicants with college degrees who have less than 12 or 24 months of work experience.” … Plaintiffs propose negotiating search terms with Defendants that target experience and recent graduate limitations…. In contrast, Defendants suggest filtering positions in their Workday database based on several data fields and then training an artificial intelligence model to identify job descriptions meeting Plaintiffs’ definition of “Recent Graduate Positions.”

Id. at *3 (emphasis added).

The court adopted Defendants’ proposal to employ the Workday database fields.  However, addressing the A.I. proposal, the court wrote:

Plaintiffs’ proposal to negotiate appropriate search terms targeting experience and recent graduate limitations is ADOPTED-IN-PART, and Defendants’ competing proposal to train an artificial intelligence model is REJECTED. Defendants contend that the Workday database does not have a field that can cleanly isolate Recent Graduate Positions, and the requirements for these positions are phrased differently, posing the risk of an underinclusive search….  Plaintiffs note that they “requested the sample of job postings Defendants reviewed in reaching that conclusion[,]” but “Defendants have not produced any sample.” … A collaborative process to agree on search terms is preferable to the unilateral training of an artificial intelligence model, which is no less likely to raise issues of under-inclusiveness. However, Plaintiffs’ proposal is REJECTED to the extent that they seek to compel Defendants to produce a 30% sample of Recent Graduate job postings across the relevant time period. Plaintiffs provide no authority in support of this proposal, which is effectively a motion to compel discovery. Therefore, IT IS ORDERED that the parties shall meet and confer on search terms to employ in the search process within fourteen (14) days of this Memorandum Order.

Id. at *4 (emphasis added). The court ordered that:

Defendants’ proposals to: (i) employ the Workday database fields identified in their supplemental submission to identify which positions quality as “Recent Graduate Positions,” and (ii) exclude from the realm of Recent Graduate Positions any positions open to recent master’s graduates or “degree equivalency” positions open to either recent college graduates or those with equivalent work experience are ADOPTED. Plaintiffs’ proposal to negotiate appropriate search terms targeting experience and recent graduate limitations to further identify Recent Graduate Positions is ADOPTED-IN-PART, and Defendants’ competing proposal to train an artificial intelligence model is REJECTED. Accordingly, IT IS ORDERED that the parties shall meet and confer to negotiate appropriate search terms within fourteen (14) days of this Memorandum Order.

Id. at *5 (emphasis added).

“Three-strata validation” was requested and rejected in In re Class Action Settlement Administration Litigation, 2026 WL 2444936 (D.D.C. Aug. 19, 2026). The parties in that case had narrowed their disputes and their “only remaining disagreement concerns how a party producing documents in discovery must validate that its productions are substantially complete.”

Plaintiffs asked for “three-strata validation.”  Defendants objected to the second and third strata.

The court explained the “three-strata” approach as:  “validation of documents coded as non-responsive, validation of documents coded as responsive, and validation of documents excluded from the universe of ESI material subject to discovery review.”

THE PARTIES AGREED TO NULL SET TESTING

The parties agreed that there is an obligation  “to confirm that responsive documents are not erroneously coded as non-responsive and omitted from their productions.” This involves review of a “statistically valid random sample” of documents coded as non-responsive. Id. at *1.

This standard practice is commonly called “elusion testing,” review of the “discard pile,” or, review of  the “null set.”  See Deal Genius, LLC v. O2COOL, LLC, 682 F. Supp. 3d 727, 734–35 (N.D. Ill. 2023) (“explaining that elusion testing, where the producing party reviews a random sample from the ‘null set’ of documents that did not hit on search terms and documents that were deemed non-responsive, is a ‘standard quality assurance practice’ that can ‘confirm the reasonableness and proportionality’ of the production”).

VALIDATION OF DOCUMENTS CODED AS RESPONSIVE – REJECTED

Defendants argued that validation of documents coded as responsive, to find false positives, was superfluous.  They asserted that Fed.R.Civ.P. 26(g) required that they certify that their response is complete and correct.  The court wrote: “Rephrased in statistical terms, producing parties must confirm that their false-negative rate falls within acceptable limits.”  In my less elegant words, a production may include nonresponsive documents; but, it cannot be a “dump truck.”

The court explained:

Review of documents marked responsive falls outside the bounds of litigants’ Rule 26(g) obligations because such review cannot identify whether documents are missing from a production, rendering it incomplete. Confirming that documents designated responsive are truly responsive reveals only the production’s false-positive rate, a proxy measure for whether the litigant has overproduced discovery material. But the Federal Rules do not prohibit parties from overproducing documents. Accordingly, this Court will not require defendants to incur additional costs re-reviewing documents that they have previously coded as responsive and intend to produce, barring claims of privilege. [emphasis in original].

VALIDATION OF DOCUMENTS EXCLUDED FROM THE UNIVERSE – REJECTED

The second strata consists of “ documents excluded from the universe of ESI material subject to discovery review.”  In other words, Plaintiffs asked for a review of documents that Defendants had not collected and that, presumably, were  not in Defendants’ litigation review platform.

The court wrote that examination of this set of documents, the “Excluded Set,” “could reveal false-negatives: responsive materials that should be produced but were erroneously omitted from the universe of materials collected for discovery review.”

However, the court responded that Plaintiffs’ request was duplicative and made too late, long after collection.  It wrote:  “Yet the Court is puzzled why plaintiffs request review of the Excluded Set after discovery review is largely complete, given that the parties also intend to review the Excluded Set for responsive material prior to beginning that review.” [emphasis in original].

The court explained that “ESI discovery generally proceeds in two phases: document collection and merits review.”  The parties had earlier “set forth agreed upon parameters for collecting the universe of documents that will be subject to merits review.”  That earlier agreement had required statistical sampling.

Insofar as the parties have agreed to statistically validate the completeness of the set of documents collected for merits review before even beginning that review, the Court finds that further validation of the Excluded Set after merits review is largely complete would be duplicative and unnecessary.

However, Plaintiffs made a different argument—so-called “substantial completeness evaluation.”  They argued that:

[T]heir proposed validation efforts “answer different questions at different points in time,” because search term validation “tests whether agreed terms are performing as intended before review begins,” while substantial completeness validation of the Excluded Set “tests whether the entire process — including all downstream coding decisions, review workflow, and production choices — resulted in a complete production at its conclusion.”

This argument was unsuccessful because the court held that “the Excluded Set is a collection of documents excluded before merits review begins and not subject to downstream coding decisions, review workflows, and production choices.”

Plaintiffs also contended that “the prevailing practice is for parties producing discovery to validate the Excluded Set both upon its creation and after merits review….”  The court noted that the authorities are split; however, it held that “the Court will not require producing parties to incur the cost and burden to re-validate the Excluded Set at the substantial completeness stage, provided that they statistically validate that documents excluded from merits review do not contain responsive material before beginning that review.”

I would add that “substantial completeness validation” of the Excluded Set could de-rail a case and mandate a “do over.”  If a sufficient number of false negatives was found in that set—and sufficiency is always debatable–presumably Defendants would have to engage in a new collection, review, and production process.

This blog was initially posted on  Electronic Discovery Reference Model

The issue before the court arose out of the developer’s motion to compel. Blue Moon made an interesting challenge to privilege based on a disclosure to the condominium’s management agent.

This blog, however, addresses the search methodology issue and Blue Moon’s assertion that Trailsedge must search all current and former unit owners’ email and text messages.  Trailsedge Owners Ass’n., Inc. v. NED Ski Associates, LLC, 2026 WL 2083546 (D. Vt. Jul. 16, 2026).

While the court rejected defendants’ untimely demand to use their search terms, it found gaps in plaintiff’s keywords and ordered broad disclosure.  The court rejected defendants’ demand to search the ESI of all current and former unit owners over more than a decade.  However, based on certain statements of plaintiff, it found that plaintiff had opened the door to additional unit owner disclosure, which was ordered by the court.

SEARCH METHODLOLGY

In my opinion, both parties were at fault when it came to search terms.  Trailsedge decided on its keywords, unilaterally and without consultation.  However, “Trailsedge argues that Blue Moon … never asked to negotiate search terms, and failed to propose its own terms until approximately two years later.”  Id. at *3.

But, “[i]n reply, Blue Moon denies it acquiesced to Trailsedge’s unilateral search process, arguing it did not know Trailsedge had used keyword search terms until many months after the discovery production and was entitled to assume Trailsedge had produced all responsive materials…. Blue Moon also argues that the scope of the requested search is proportional because Trailsedge seeks damages exceeding $10 million for alleged defects dating back to the period when the buildings were constructed and occupied while refusing to identify the date range, custodians, and results of its searches.”  Id. at *4.

The court looked to the Sedona Conference, id. at *4, passim, and  Judge Peck’s decision in William A. Gross Constr. Assocs., Inc. v. Am. Mfrs. Mut. Ins. Co., 256 F.R.D. 134 (S.D.N.Y. 2009), writing that “when a party uses keyword searches to identify responsive ESI, the search process must be cooperative, well-informed, capable of explanation, and sufficiently transparent to permit the opposing party and the court to evaluate it if challenged.”  It wrote that cooperation and proportionality are the dominant drivers. “As Gross explained, keyword searches must be developed with input from custodians who know the terminology used in the relevant communications and must be tested to ensure accurate retrieval and reduce false positives.” Id. at *5.

The Trailsedge court continued:  “Applying the principles described above to this case, the court first rejects Blue Moon’s request to impose its proposed keyword search” on Trailsedge.  Id. at *5.  It found that those terms may be overbroad, redundant, “or likely to generate excessive false positives.”  Further, Blue Moon was untimely:

Sedona’s principles do not give a party license to wait until late in discovery and then demand a wholesale redo of its opposing party’s collection process using its preferred terms….  Trailsedge also notes that Blue Moon did not request search-term information or propose its own search terms until approximately two years later…. The timing and proportionality considerations weigh against granting Blue Moon’s request in full.

Id. at *5.

However, Trailsedge was not off the hook with its unilateral selection of keywords. It wrote that: “The court… also rejects Trailsedge’s argument that Blue Moon’s delay ends the inquiry. Trailsedge chose to use keyword searches as part of its collection process, and the reasonableness of that process became a fair subject of discovery once it was challenged.” Id. at *6.

Again citing Sedona and Judge Peck, the Trailsedge court noted “the more basic principle that counsel must understand, supervise, and be able to describe the search process used to identify responsive ESI in enough detail to assess its adequacy.” Id. at *6.  Finding “apparent gaps” in the terms designed unilaterally by Trailsedge, the court wrote:

Trailsedge has not met its obligation to provide enough information to permit a meaningful assessment of whether the searches were reasonable and proportional. The record does not show that Trailsedge has identified the custodians searched, the date ranges applied, the dates the searches were conducted, the terms used for each custodian or source, or the results returned by each term….  Trailsedge must therefore provide additional information about its search methodology.

Id. at *6.  It added that Trailsedge’s  “omission of central party names and words related to critical legal issues raises a sufficient question about the adequacy of Trailsedge’s prior search sufficient to warrant additional discovery.”  Id.

Accordingly, within fourteen days, Trailsedge shall serve on Defendants a written description of its prior ESI search methodology. That description shall identify: (1) each custodian whose emails, text messages, or other ESI were searched; (2) each data source searched; (3) the date range applied to each search; (4) the date on which each search was performed; (5) the search terms used for each custodian or source; and (6) the number of hits returned by each term (to the extent reasonably available). Trailsedge shall also state whether custodians were instructed to use particular search terms, whether custodians self-selected additional terms, and what steps counsel took to supervise or validate the searches.

Id. at *6.  This was to be followed by a court-ordered “meet and confer.” Id.; see generally, the Hon. Paul W. Grimm (ret.), Michael D. Berman, et al., “Discovery about Discovery: Does the Attorney Client Privilege Protect All Attorney-Client Communications Relating to the Preservation of Potentially Relevant Information,” 37 U. Balt. L. Rev. 413 (2008).

SEARCHES OF EVERY CURRENT AND FORMER UNIT OWNER

The court explained the unit owner issue as follows:

Blue Moon seeks an order requiring Trailsedge to collect and produce responsive documents from all current and former unit owners for the period of January 1, 2011 through May 12, 2023…. Blue Moon argues Trailsedge was required to search all unit owners’ files because Trailsedge “is suing on behalf of all unit owners” and because Trailsedge’s initial disclosures allegedly identified members, not just Board members, as persons likely to have discoverable information.

Id. at *7. “Trailsedge responds that the Association is a Vermont nonprofit corporation distinct from its members and that the unit owners are not parties or custodians of Association records merely because they own units…. Trailsedge further argues that, except in limited governance matters, the Association acts through its executive board under the Vermont Nonprofit Corporation Act and that the Association’s statutory authority to litigate in its own name on behalf of itself or unit owners does not collapse the distinction between the Association and its members….  Trailsedge also argues that the request is disproportionate because Trailsedge consists of twenty-four units that have been bought and sold over the years by an unknown number of owners and Blue Moon seeks documents from January 1, 2011 through May 12, 2023.”  Id. (emphasis added).

The court resolved the issue by applying the “possession, custody, and control” and proportionality  standard of Fed.R.Civ.P. 34.  “In the District of Vermont, ‘control’ is construed broadly and is not limited to legal ownership or actual physical possession; documents may be within a party’s control when the party has the ‘right, authority, or practical ability’ to obtain them.”  Id. at *7.

It held that Blue Moon had the burden of proving control by Trailsedge and failed to carry it: “The record also does not support treating all unit owners as custodians simply because some unit owners may possess relevant information. Rule 34 control requires more than relevance.”  Id. at *7.  The court noted that the Sedona conference calls for “caution” in evaluating information in the possession of non-parties.

Although the [Sedona Conference] Commentary’s recommended “legal right” standard is narrower than the “practical ability” standard applied by district courts in the Second Circuit, it identifies generally applicable practical and fairness concerns implicated when discovery obligations are extended to ESI held by third parties. The Commentary explains that an expansive approach may require parties to preserve, search, and produce documents they do not actually possess and may not have the ability or legal right to obtain…. That concern is present here. Blue Moon has shown that unit-owner communications may be relevant, but it has not shown that Trailsedge has actual possession, legal authority, or a practical ability to obtain every current and former unit owner’s personal emails and text messages on demand.

Id. at *7 (emphasis added).  The court also evaluated proportionality: “The documents Blue Moon seeks may be relevant to notice, knowledge, and statute-of-limitations issues. Blue Moon’s requested owner-wide search, however, would encompass more than twelve years, cover all current and former owners of twenty-four units, and include potentially sensitive personal texts and emails.”  Id. at *8.

However, Trailsedge was not off the hook. Its discovery responses “reasonably could have led Blue Moon to believe that at least some non-Board Association members had supplied documents or information.” Id. at *8.  In my words, Trailsedge opened the door.  Therefore:

In light of the ambiguity created by Trailsedge’s interrogatory responses, however, the court finds the responses require clarification. Rule 26(g) requires Trailsedge’s discovery responses to be formed after a reasonable inquiry, and Rule 26(e) requires supplementation if the responses were materially incomplete or incorrect. Fed. R. Civ. P. 26(e), (g). Because Trailsedge’s responses suggest that some member-produced materials were considered but do not identify which members provided documents, what sources were searched, or whether any non-Board member communications were requested or reviewed, Trailsedge must clarify those issues in the search-methodology disclosure ordered in the previous section. The disclosure shall state whether Trailsedge requested documents from any non-Board unit owners; identify any non-Board unit owners from whom Trailsedge obtained documents or information used in preparing its discovery responses or productions; identify the categories of documents obtained from those members; and certify whether Trailsedge has produced all responsive, nonprivileged documents from those member-provided materials that are within its possession, custody, or control.

Id. at *8.

For more on “possession, custody, and control,” see:

• Possession, Custody, or Control – Need for a Uniform National Standard – Part II;
• Isn’t It Time for a Uniform National Standard on “Possession, Custody, or Control”?

1. The Parties agree that each Responding Party is best situated to determine the most appropriate method or methods for that Responding Party to search, collect, cull, and produce documents responsive to discovery, consistent with Sedona Conference Principle 6 which instructs that “[r]esponding parties are best situated to evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information.” Nothing in this ESI Protocol alters a Responding Party’s obligation to conduct a reasonable inquiry or use a reasonable process in searching for and producing relevant information, nor does it alter a Requesting Party’s right to challenge a deficient production.
2. The Parties will meet and confer to discuss the use of search terms, filters and date ranges or the use of advanced search and retrieval technologies. Within a reasonable time period, Responding Party shall specify the technique or techniques (e.g., search terms, technology assisted review (“TAR”), artificial intelligence or machine-learning technology (collectively, “AI”)) it will use to search for and cull material it reasonably anticipates will be responsive to Requests for Production. If the Requesting Party objects to the particular technique or techniques, the Parties will timely meet and confer regarding the matter and submit any unresolved disputes to the Court for resolution.
3. A Party who intends to use search terms will describe a Search Protocol that will include: (a) the criteria to be used to identify the universe of Documents to which search terms will be applied (e.g., date range), and (b) proposed search terms to be applied to that universe of Documents, subject to revision based on, for example, meet and confer of the Parties and the number of hits returned by each term. A Requesting Party may also suggest search terms to be applied. Each Requesting Party shall limit its ESI production requests to a total of twenty search terms per custodian per party, for the avoidance of doubt a custodian can include either Party itself for searches of Party-wide electronic repositories. The Parties may jointly agree to modify this limit without the Court’s leave. The Court shall consider contested requests for additional search terms per custodian, upon showing a distinct need based on the size, complexity, and issues of this specific case. The search terms shall be narrowly tailored to particular issues. Indiscriminate terms, such as the producing company’s name or its product name, are inappropriate unless combined with narrowing search criteria that sufficiently reduce the risk of overproduction. The Parties acknowledge that “indiscriminate terms” would not include names of various publicly available training datasets such as, “Books3.” A conjunctive combination of multiple words or phrases (e.g., “computer” and “system”) narrows the search and shall count as a single search term. A disjunctive combination of multiple words or phrases (e.g., “computer” or “system”) broadens the search, and thus each word or phrase shall count as a separate search term unless they are variants of the same word or identify alternative names, examples, or members of the same discrete set of datasets, websites, or repositories. Use of narrowing search criteria (e.g., “and,” “but not,” “w/x”) is encouraged to limit the production.
4. Any Search Protocol will include a requirement for the Responding Party to provide hit reports for proposed search terms prior to utilizing the search terms to narrow the universe of Documents to be reviewed. Search term hit reports will include, for each term, the number of documents with one or more search term hits, that number plus any attachments to those documents, and the number of documents hitting uniquely on that term and not on any other term in the same list. The report will also include total number of documents hitting on one or more terms and that number plus any attachments to those documents (presumptive review population). Any Search Protocol shall include a requirement to review one random statistical sample of distinct Documents, determined by a sampling of a confidence level of 95% and a margin of error of 2%, that do not hit on the search terms that are agreed upon by the Parties (the “Null Set”). The Null Set of distinct Documents shall be generated after all document processing is substantially complete. These Null Set Documents should come from each custodian and include Documents throughout the entire date range. The Responding Party should then review the Null Set Documents and include in future productions any responsive Documents located therein. If more than 10% of the Null Set are responsive, then the Parties shall revisit the agreed-upon search terms by analyzing the Documents produced and determining which additional terms are necessary. The Parties agree to meet and confer regarding the Search Protocol and to raise any disputes regarding the Search Protocol for resolution by the Court.
5. Nothing in this order or any other document relating to discovery in this matter shall be construed or interpreted as precluding a Responding Party from performing a responsiveness review to determine if documents captured by search terms are in fact responsive to the requesting party’s request. Further, nothing in this order or any other document relating to discovery in this matter shall be construed or interpreted as requiring the production of all documents captured by any search term if that document is – in good faith – deemed not responsive to the requesting party’s request for production by the Responding Party.
6. If a Responding Party elects to use search terms in combination with TAR on the same set of Documents (“layering”), it shall disclose that intent to the Requesting Party before doing so, and the Parties shall meet and confer on a protocol governing the use of layering. The protocol shall include a method to compare the number of hits when there is and is not layering for the set of Documents at issue, and permit the Requesting Party to make a reasonable request that the Responding Party review a reasonable sample of Documents excluded from review through the layering process to assess whether responsive Documents have been excluded. In the event that the Parties cannot reach an agreement on a protocol after meeting and conferring, the Parties shall submit competing protocols to the Court.
7. A Party who intends to use TAR to reduce the population of documents to be reviewed shall disclose a TAR Protocol that includes, at a minimum, the information listed in Appendix 3.
8. A Responding Party that elects to use AI to assist in responsiveness or privilege determinations (“AI Responsiveness Review”) shall disclose that election to the Requesting Party. AI Responsiveness Review constitutes workflows utilizing large-language models (LLMs), deep-learning classifiers, embedding-based similarity analysis, semantic clustering, predictive redaction systems, or generative summarization models used to determine whether a Document will be produced, withheld, or redacted.
9. The Responding Party shall provide an AI Review Protocol, which shall include, at a minimum, the information listed in Appendix 4.

Nothing in this Section shall be construed to diminish the Responding Party’s obligations under the Federal Rules of Civil Procedure, this Order, or applicable law. Legal counsel remains responsible for ensuring the accuracy and completeness of all productions.

Documents or ESI known to be responsive to a discovery request or relevant to the subject matter of this action shall be produced without regard to whether it was responsive to any search methodology described herein or developed in accordance with this Order, unless Counsel specifically identifies the documents being withheld and provides a specific objection for withholding each withheld document.

Documents which are reasonably believed to be responsive and for which text-based search technologies are fundamentally ineffective, such as images, video, certain spreadsheets, certain hard copy documents, certain documents from noncustodial sources, or certain foreign language documents where the Parties do not have suitable search terms in such language, must be reviewed without culling by search terms, predictive coding, or other technologies that rely primarily on text within the document. Prior to the production of such unsearchable items, the Responding Party may conduct a page-by-page review for responsiveness, confidentiality, privilege, and other protections.

Clearly the court is privy to information I do not have and may have explained its reasoning in greater depth elsewhere. For example, the opinion opens by stating: “For the reasons discussed during today’s discovery conference, the parties’ current discovery disputes are resolved as specified below.”

ORDER THAT EVERY “HIT” MUST BE PRODUCED, UNLESS PRIVILEGED

First, the court ordered plaintiff Sussman to run defendant VWR’s two disputed search strings across all of its custodians:

Documents from October 1, 2018 to the present: Whiteboard AND NOT (*@whiteboardadvisors.com OR “whiteboard advisors” OR Edmentum OR “Paul Joy” OR “GSV’s News” OR “@morningbrew.com” OR “Carr (meeting rooms)”)

Documents from January 1, 2015 to the present: (Dwayne OR Dwayne’s) OR dwaynej*

It then ordered that “Sussman must produce the (i) hit reports for the searches specified above and (ii) all of the documents that hit on the search terms, except those (if any) withheld for privilege.” Id. at *1 (emphasis in original).

The court added: “For avoidance of doubt: Sussman may not withhold documents hit on by the two modified search strings set forth above based on a responsiveness review.”   Id. (emphasis added).

It may be that the court had sufficient information to determine responsiveness based on the detailed search string.

But I support the general rule that the fact that a document “hits” on a search term or string does not mean that it is responsive.  And, Fed.R.Civ.P. 26(b) limits the scope of discovery to information related to claims and defenses.  Non-responsive information fails to meet that standard.

For additional authorities, please see:

• Does Every “Hit” on a “Hit Report” Have to be Produced? (Nov. 13, 2024);
• How to Avoid Contentious “Hit Report” Problems – Part III (Nov. 15, 2023);
• How to Avoid Contentious “Hit Report” Problems (Jan. 4, 2023);
• Hit Reports (Aug. 14, 2022).

Best practice would be for the parties to cooperate and agree in advance.  Craig Ball posted Federal Court Rules on Whether Documents Containing Agreed-Upon Keywords are Responsive Per Se | Ball in your Court (Oct, 25, 2021). He wrote:

If you enter into an agreement with the other side to use keywords and queries for search, be clear about expectations with respect to the disposition of items hit by queries.  Assuming the items aren’t privileged, are they deemed responsive because they met the criteria used for search or is the producing party permitted or obliged to further cull for responsiveness based on the operative Requests for Production?  You may think this is clear to the other side; but, don’t count on it.  Likewise, don’t assume the Court shares your interpretation of the protocol.  Just settling upon an agreed-upon list of queries may not be sufficient to insure a meeting of the minds.

I suggested model language in Hit Reports:

In the spirit of cooperation and to further the goal of proportionality, we agree to run one or, within reason, more “hit reports” for you.  The purpose is to permit you to focus your subsequent requests.  We will use [name of litigation review software] and run the terms against electronically stored information in the [name database].

Please note that a “hit” may be beyond the scope of discovery, irrelevant, privileged, work product, non-responsive, a trade secret, confidential, proprietary, etc., and all rights are reserved to object to the production of any or all “hits.” This includes without limitation the right to assert that any or all production is not proportional to the needs of this case.  Further, all rights are reserved as to any and all other objections under the Maryland Rules [or Federal Rules of Civil Procedure], including the right to move for a protective order for any reason permitted under the Rules. All objections to any discovery request are incorporated herein, and none are waived by the hit report. No privilege is waived by production of a hit report because the hit report is not an offer to produce any document. Additionally, the right to mark a “hit” as confidential or highly confidential under the agreed protocol or protective order is fully reserved. Further, we reserve the right to add additional data to, or remove data from, the database and that may vary the number of hits.  All rights are reserved, including without limitation, rights under Fed.R.Evid. 502.

It may be that, in Sussman, the detailed search string minimized the risk of production of non-responsive documents; however, the scope of the order on its face appears to go beyond the scope of discovery by ordering production of non-responsive ESI.  For example, the second search string covers all Sussman custodians from 2015 to 2026.

WAS IT ERROR FOR SUSSMAN TO PRODUCE

MANY DOCUMENTS THAT DID NOT CONTAIN AGREED-UPON SEARCH TERMS?

Second,  Sussman said that it ran the agreed-upon search terms and produced approximately 800 pages.  However, “VWR states that the June 28 production cannot be what Sussman claims it to be, because many if not all of those 800 documents do not contain any of the agreed-upon search terms.”  Id. at *1 (emphasis added).

The court added that “Sussman has done little or nothing to verify the content of its production (or supplement it if necessary)….”  It ordered  that Sussman “consult with its discovery vendor as to what search terms were actually run in connection with its June 28, 2026 production” and “if necessary, run the correct searches, using the agreed-upon terms….”

The parties agreement is silent on whether the documents produced were parent-child documents.  If they were, it is not uncommon that a search term “hits” in one parent or child, but production of the parent-child email results in production of one or more documents that do not contain the search term.

I agree that explanation may not provide Sussman with a “safe harbor” here because “many if not all” of the produced documents did not contain the keywords.  And, the court noted a lack of quality control by Sussman, i.e., “has done little or nothing to verify the content.” But a little more information would be helpful.

There are some other factors that may have contributed to the court’s decision. Sussman’s production was tardy.  For example: “The Court notes that this information was due by May 14, 2026, and Sussman is now more than two months late.”  And, as to other information: “The Court notes that this information was due by March 6, 2026, and Sussman is now more than four months late.”  It may be that it was time for the curtain to fall.

This blog was initially posted on  Electronic Discovery Reference Model.

Ms. Kim sued, alleging pregnancy discrimination.  She had been terminated upon return from maternity leave. Defendants asserted that termination was part of a reduction in force.  Ms. Kim alleged that the reason was pretextual.

This post focuses on only the Teams issue. The Kim court wrote: “Plaintiff argues that Defendant’s searches for relevant electronically stored information (ESI) were deficient. As it turned out, Defendant had not searched Microsoft Teams. But MS Teams, as one of Defendant’s own witnesses (Kathryn Raney) testified, was one of the primary communication methods used at Cushman & Wakefield.”  Id. at *1.

The “oversight” was discovered and “[t]o Defendant’s credit, upon discovering the MS Teams gap, counsel immediately ran the existing search terms against MS Teams and produced 47 pages of messages” one day after the discovery.  Two of the newly-produced messages were relevant to the pretext analysis. Id.

Understandably, the Plaintiff asked for more—too much more: “Plaintiff asks for a nearly indiscriminate search of ‘all reasonably likely repositories.’”

Defendant offered much less: “Defendant says it already ran the search terms against Teams and there’s nothing left.”

The court said: “Neither position is quite right.”  It then turned to the specifics:

The problem with simply rerunning the existing search terms on Teams is that those terms all require “Connie Kim” as an anchor—e.g., “Connie Kim” NEAR “terminat!” …. It is arguable whether that may work well enough even for emails, but it cannot work for MS Teams chats about transition planning among managers who might say “the Smartsheet” or “Brooke’s workload” without mentioning Plaintiff by name. Keyword searches alone, without more advanced and thoughtful search techniques, will be inadequate for Teams data—a medium where conversations are shorter, more informal, and less likely to include full names than email.

Id. at *2 (emphasis added).

Faced with divergent positions following a defective search, the court said:

That does not mean, however, that Plaintiff gets to enforce the broadest version of a document request—seeking “all documents and communications relating to Plaintiff or her allegations”—or to demand an expansion of custodians or data sources with no concern for proportionality. At the same time, overbroad requests do not excuse incomplete searches. An objecting party that elects to search and produce—rather than move for a protective order—undertakes an obligation to search reasonably. See Fed. R. Civ. P. 26(g)(1)(B). Defendant just didn’t search everywhere it should have. And the proportionality analysis under Rule 26(b)(1) weighs in Plaintiff’s favor with respect to MS Teams. This is a pregnancy discrimination case in which the timing and circumstances of the termination decision are among central disputed facts. Defendant has exclusive access to its own internal communications about termination decisions before any final termination news was delivered to Plaintiff. The transition-related Teams messages already produced confirm that relevant communications exist in this repository.

Id. at *1 (emphasis added).

The Kim court entered alternative holdings.

The first holding was:

Ultimately, the court is in no position to prescribe how Defendant must conduct the supplemental search of MS Teams. Defendant is free to consult with its eDiscovery consultant on the most efficient and defensible methods for searching, reviewing, and producing Teams data. What this order can prescribe, though, is the scope of the obligation. How Defendant fulfills that obligation—whether through custodian-based collection, refined keyword queries, or technology-assisted review—is Defendant’s choice, so long as the search is reasonable and the production is complete.

Id. at *2.

The alternative holding was that absent agreement, three named custodians were to be searched using a date range specified by the court, and under a judicially-specified scope.  The court added: “Nothing in this order prevents the parties from cooperating in the refinement of search parameters for MS Teams. But absent agreement, the scope articulated above shall govern.”  Id. at *3.

The court reasoned that the cause of the problem was failures by both parties during the Fed.R.Civ.P. 26(f) conference:

To the extent that the burden of additional searches creates work for Defendant, that is in large part a problem of their own making by not including a primary communications platform like MS Teams in their discovery plan. This court’s Standing Order on Civil Discovery—which has been in effect throughout this case—emphasizes the obligation to identify ESI sources, repositories, and communication platforms during the Rule 26(f) conference. …. Had Defendant identified Teams as a communication platform at the outset, the search would have been completed months ago and no compressed deadline would be necessary. At the same time, Plaintiff shares much of the responsibility, too. The Joint Rule 26(f) Report … reflects a single paragraph on ESI: the parties are “unaware of any e-discovery issues” and agree to “cooperate to develop search terms, protocols, and methods to minimize the expense of ESI.” That is the kind of pro forma ESI discussion this court’s Standing Order warns against.… Had the parties conducted a substantive ESI conference at the outset—identifying repositories, custodians, and communication platforms as the Standing Order and the 2006 Advisory Committee Notes to Rule 26(f) require—the MS Teams gap would have been identified months ago and this motion wouldn’t have been necessary. [emphasis added].

I have some concerns.  First, of course, every case is dependent on  its facts.  However, I can envision many cases where one size will indeed fit all, email and Teams, including, for example, chats.  Second, Teams may, for a non-exhaustive example, contain audio recordings, meeting records, and transcripts, none of which appear to be either short or informal.

I asked Tom O’Connor of Gulf Coast Legal Technology Center for some of his thoughts.  In response to the court’s suggestion that keyword searches “without more advanced and thoughtful search techniques, will be inadequate for Teams data,” he explained that keyword searches inside Teams work. But Teams supports only basic keyword matching and a few command-style filters. This means that Outlook can be precise, while Teams requires looser terms to catch variations.

He raised another important point—what tool was used for the search? Was it a search of Outlook email or Teams chats?  Was Purview or a third-party tool used?

Tom explained:

Teams search indexes chat differently than email.  It:

• Prioritizes exact word matches;
• Does not index message metadata as richly as Outlook;
• Often misses partial-word matches; and
• Returns fewer results when the term is too specific.

Of course, it is undoubtedly correct that the Boolean logic employed—whether “Connie King” is a  workable anchor–must be properly designed.

This blog was initially posted on  Electronic Discovery Reference Model and  JD Supra.

After some back-and-forth negotiations, “Defendants notified Ms. MacDonald that the agreed-upon search terms would not be workable because they hit on approximately 3.1 million documents totaling 2,547.6 gigabytes of data…. Defendants indicated that processing and hosting that data would cost $620,000 and attorney review costs would exceed $3.1 million.”

Defendants proposed another conference of the parties. “Ms. MacDonald refused.” After Defendants made a small production, Ms. MacDonald moved to strike their objections, compel responses, and for sanctions.

After addressing some preliminary issues, the court wrote:

The crux of the parties’ disagreement as to many of the discovery requests hinges not on the relevance of the documents sought but rather on the burden of locating responsive ESI. The parties have engaged in a needlessly ineffective and prolonged debate over search terms without first identifying the concrete universe of potentially responsive information. A list of terms that returns over 3 million potentially responsive documents is unworkable for both parties in terms of time and expense. A better approach would have been for the parties to first agree to custodial and non-custodial sources of documents, establishing a total universe of ESI. Defendants’ eDiscovery vendor then could have run every list of proposed search terms against the entire universe of documents in order to identify where terms were overbroad and unlikely to result in responsive information. The parties could then negotiate over connectors and other Boolean terms in order to further narrow the results. Utilizing these so-called “search term hit reports” would create full transparency into the resulting universe of potentially responsive documents and thus allow the parties to establish a list of search terms that would be workable for both sides. All of this could have occurred before the parties utilized the Court’s time and resources to have two discovery dispute conferences at which they purportedly agreed to a list of search terms despite not having even a basic understanding of how many documents those search terms would return. [emphasis added].

After addressing proportionality, the court added: “The burden and expense required for Defendants to review over 3 million documents are not proportional to the needs of this case, to say nothing of how significantly such an undertaking would delay the case. It has already been nearly a year since discovery began. The Court is left with no choice but to order the parties to reconvene their negotiations over search terms and strongly urges the parties to utilize search term hit reports to make those discussions more efficient and effective.”

The court called for a “mutually agreeable ESI protocol,” and wrote that “[t]he parties have already agreed to custodians and are now ordered to agree to a list of mutually agreeable search terms—that hit on a workable number of documents for both parties….”  It added: “If the parties are unable to agree, they may request an informal discovery conference with the Court. If a conference is needed, Defendants should be prepared to provide specific data supporting which search terms are overbroad.”

In similar situations, some courts have directed that parties either record or transcribe their meeting. If agreement is not reached, the court may review that record:

• “Meet and Confer” Was Transcribed (Dec. 1, 2025);
• Felder: Part 1 of 4:  Court-Ordered Transcription of “Meet and Confer” Session (Aug. 13, 2024)(Court orders for recording of “meet and confer” sessions has a long pedigree, going back to at least 2008. My first recollection of hearing about such an order came from a presentation by the late Hon. David J. Waxse);
• But cf. How Not to Negotiate an ESI Protocol? Say it is “Mandatory”; and, Demand That Discussions be Recorded (Mar. 26, 2025).

The City of Ann Arbor had denied employees’ requested religious exemptions from its mandatory vaccination policy during the COVID-19 pandemic.  They sued. Due to the high volume of discovery disputes, the court appointed a retired Magistrate Judge as a discovery master. This decision is on objections to the master’s decisions.

THE DISCOVERY MASTER’S DECISION

The court set the stage, writing:

Plaintiffs argued that the City’s production in .PDF format was not compatible with their counsel’s document review platform, and they wanted the documents produced in another format[1] at no cost to them. Plaintiffs also contended that the City included junk documents with the production (e.g., newsletters from third parties advertising deals for laptops and vacations).

The discovery master held that the production was in accord with Fed.R.Civ.P. 34(e) and the parties’ discovery plan.  The master held “that Plaintiffs failed to show that the ‘junk’ documents were not in fact responsive to their RFP…..” She also noted that production began in March 2024 and plaintiffs did not raise objections until February 2025.

THE NATIVE FORM ISSUE

Plaintiffs “claim that the parties’ discovery plan allowed them to request documents in their original or native format, and that they did just that, but the City refused to produce documents in native format unless Plaintiffs paid significant costs.”

However, the court disagreed, writing: “As [the discovery master] correctly found, Plaintiffs’ RFP did not request documents in their original or native format, even if the discovery plan enabled them to do so.”  The court continued:

In fact, the word “native” appears nowhere in the RFP, and the word “original” appears only in the definition of “Document” to include copies differing from the original….  The language from the RFP that Plaintiffs quote to show that they asked for documents in their native or original format did not do so…. Instead, it simply instructed the City to not engage in a “document dump,” and to “categorize the documents and identify which discovery requests documents are responsive to for [sic] all documents produced.”

The City made four productions, “electing to produce documents as searchable, bookmarked PDFs compatible with Adobe, which was in accordance with the agreed upon protocol. Even if the bookmarked PDF did not work with Plaintiffs document review platform, they acknowledge that the City’s first and fourth document productions are organized using embedded bookmarks correlating to the relevant document requests.”

PRODUCTION OF “JUNK”

Plaintiffs complained of the production of “junk” documents, such as newsletters from third parties advertising deals for laptops and vacations.  “According to Plaintiffs, [the discovery master] failed to address their complaints concerning thousands of blank pages with deleted or missing information, including thousands of blank images in emails and pages with no content which were included in the City’s production.”

The Alexa court rejected plaintiffs’ argument, writing:

[Plaintiffs] fail to show that the documents were not responsive to the RFPs. The documents produced in response to Plaintiffs’ broad requests may in fact be irrelevant; however, Plaintiffs must show that they are not responsive to what was requested. For example, a request for “all documents, communications, and ESI from any custodian who may be identified on [the City’s] Witness List … that contain the word ‘COVID’ any time since August 1, 2021” …—notably, at time when the world was responding to the COVID-19 pandemic—was likely to generate a response containing thousands of irrelevant documents. However, that is the result of how Plaintiffs formulated the request, not how the City responded to them. [emphasis added].

The objections were overruled. The court also addressed privilege issues, including an argument of waiver by untimely logging, and other issues.  The court applied the “legal right” test to text messages on BYOD cell phones.

On the issue of “document correlation” under Rule 34(b)(2)(E), please see:

• Does Rule 34(b)(2)(E)(i) Mandate “Document Correlation” When ESI is Produced and, If So, Does Metadata Provide a “Work-Around” to Avoid the Costs of That Correlation Process? (Jul. 20, 2021);
• Document Correlation Under Fed.R.Civ.P. 34(b)(2)(E) (Jul. 29, 2025);
• Document Correlation (Nov. 5, 2025).

On the issue of possession, custody, and control and the “legal right” test, please see:

• Isn’t It Time for a Uniform National Standard on “Possession, Custody, or Control”? (Sep. 7, 2022);
• Possession, Custody, or Control – Need for a Uniform National Standard – Part II (Feb. 18, 2026).

On the issue of “junk,” please see An Epilog:  4 Things You Never Wanted to Hear + 8 Lessons & a Dispute Over Revealing Recipients of Litigation Hold Notices, All in One Case (May 10, 2024)(“Plaintiffs requested this language and thus Plaintiffs cannot be heard to complain later about receiving a large volume of system or junk files (and should not be expected to argue for extensions of the case or discovery schedule on this basis). By arguing for this language, Plaintiffs knowingly undertook the risk that they may receive a large volume of system or junk files as a result.”); 4 Things You Never Wanted to Hear + 8 Lessons & a Dispute Over Revealing Recipients of Litigation Hold Notices, All in One Case (May 4, 2024).

This blog was initially posted on  Electronic Discovery Reference Model.

____

[1] The Federal Rules of Civil Procedure use the term “form” or “forms,” not “format.”

The starting point was the court’s observation that: “Although it is abundantly clear from the parties’ filings that they have attempted to cooperate to ensure proposed discovery is consistent with the principles of relevance and proportionality, despite months’ of effort, they have resolved relatively few of their disagreements about the appropriate nature and scope of discovery in this action.”

Numerous discovery disputes were presented in this apparently massive case.  They included whether discovery was limited to the forum District, the relevant time period, the relevant sources, and proportionality.

This post focuses on “hit reports.” The court explained the opposing positions:

Because the parties have clung to materially divergent positions on the appropriate nature and scope of discovery, they not surprisingly have been unable to reach consensus on protocols to govern discovery of ESI, including the breadth of proposed search terms and the related subject of hit reports, as well as the scope of custodians to be searched.

Whereas Plaintiffs argue the Court should compel Defendants to comply with the “expected and customary” practice of providing hit reports for their requested but disputed search terms/strings…, Defendants argue they should not be compelled to provide hit reports for Plaintiffs’ facially overbroad proposed search terms….

Id. at *6 (emphasis added).

The United Farm Workers court rejected a bright-line right to “hit reports,” writing that: “Contrary to Plaintiffs’ characterization of a purported ‘expected and customary’ practice, there is no one-size-fits-all approach to managing the discovery of ESI.”  Id. at *6.

The United Farm Workers court recognized the value of “hit reports,” writing: “In many circumstances, providing early hit reports can assist the parties narrow disputed search terms.” Id. at *6.

However, the United Farm Workers court quoted another decision limiting the right to “hit reports.” That other court wrote:

[T]he Court is concerned that requiring Respondents to run twenty search terms across four custodial files, and then review and produce responsive documents, will be unduly burdensome. Further meet and confer efforts are necessary for the parties to identify appropriately narrow search terms. To facilitate this, Respondents shall provide hit reports listing the number of documents across the four custodians that hit on Harbour’s fourteen proposed search terms.

Id. (emphasis added). The United Farm Workers court also cited precedent stating:

However, where the disputed search terms are facially overbroad, a requesting party has no entitlement to early hit reports…. The court agrees that the search terms and date range are overbroad and will result in an unduly burdensome number of irrelevant hits.

Id. at *6 (emphasis added).  It then reasoned:

Consistent with the discussion above concerning proportionality, the undersigned agrees with another court’s observation that search terms “do not define proportionality. … Rather, proportionality must be assessed by considering the specific context of the custodian and/or repository against which the search terms will be applied.” … Applying that principle here, while some of Plaintiffs’ proposed search terms and strings are facially overbroad, other search terms and strings, if applied across a narrow universe of custodians most likely to possess relevant information, can more reasonably be expected to return results that are proportional to the needs of the case, consistent with Rule 26(b).

Id. at *6.

The United Farm Workers court then applied these principles:

Defendants already have run narrow search term queries across 22 custodians largely associated with USBP’s El Centro Sector (including Defendant Bovino)….  As a starting point, to facilitate continued meet/confer efforts between the parties to reach consensus on ESI discovery, the Court will order Defendants to conduct a further ESI search and collection run and produce hit reports for these 22 custodians using, in addition to Defendants’ agreed search terms, Plaintiff’s proposed search terms Nos. 1, 6, and 12…. Although Defendants assert that various limitations of their e-discovery platforms would make any overbroad ESI search requirement overly burdensome, including a significant adverse system limitation that would be triggered if any collection project exceeds 5-terabytes, nothing in the supporting declaration of U.S. Customs and Border Protection eDiscovery Team Lead… leads the Court to assess that the search and hit report requirement it orders herein will exceed that 5-terabyte threshold. [emphasis added].

I find United Farm Workers a bit confusing.  If the concern was additional collection and production, I understand it.

However, in my experience, “hit reports” are never burdensome because collection is a separate discussion and “hit reports” do not require production of anything.  “Hit reports” are easy to generate.  Hit Reports (Aug. 14, 2022).

Absent an Agreement in ESI Protocol to Produce All “Hits,” Without Review, (Mar. 4, 2025), which I suggest is almost universally imprudent, it is not necessary to produce all documents that are on a “hit report.”  See:

• Hit Reports (Aug. 14, 2022);
• Does Every “Hit” on a “Hit Report” Have to be Produced? (Nov. 13, 2024);
• Hit Reports – Part IV – Request for “Nonresponsiveness” Log Denied (Jan. 2, 2025);
• 8 Lessons Learned – Part II – “Hit” Report Provision of ESI Protocol (May 7, 2024);
• How to Avoid Contentious “Hit Report” Problems – Part III (Nov. 15, 2023).

In fact “hit reports” may have greater value when there is an assertion that search terms are too broad.  One court wrote that “any objection to a proposed search term as overbroad or disproportionate must be accompanied by a hit report so that the other party—and the Court—can assess the burden. Making such an objection without the hit report invites disputes and Court intervention, which an ESI Protocol is designed to avoid.”  What is the Purpose of an ESI Protocol? Court Addresses Seven Key Issues (Nov. 29, 2025).

This blog was initially posted on  Electronic Discovery Reference Model and  JD Supra.

One significant issue was an order limiting the use of artificial intelligence to analyze protected materials produced by an opponent in discovery.  I have suggested that A.I. Protective Orders Are Becoming Routine (Apr. 20, 2026)(citing several examples), and ByHeart is another example.

Defendants proposed “targeted modifications” to the court’s model protective order. They asserted that highly sensitive personal and commercially competitive information would be produced.  One proposed modification addressed “appropriate limitations on the use of artificial intelligence tools to analyze protected materials….” Id. at *1.  The court adopted that proposal.

The order is not reprinted in Westlaw. However, it is available on Bloomberg In Re: ByHeart, Inc., Infant Formula Marketing, Sales Practices, and Products Liability Litigation, Docket No. 1:26-md-03178 (S.D.N.Y. Apr 02, 2026), Court Docket, by subscription. In part, it stated:

The parties agree that individuals or entities with access to Protected Materials are prohibited from inputting documents, ESI, load files, and other materials produced pursuant to this Order into generative artificial intelligence tools or similar applications unless the tools or applications provide the following protections against inadvertent disclosure of data: (1) confirmation that the tool or application will not train the model or incorporate into model weights any inputs containing those Protected Materials or otherwise inadvertently produce outputs containing identifiable parts of Protected Materials; (2) administrative controls to allow deletion of Protected Materials by the user and to prevent unauthorized access to Protected Materials; and (3) adherence to cybersecurity best practices

Document 45, Docket No. 1:26-md-03178 (S.D.N.Y. Apr 23, 2026), ¶20.  Is this now the standard of care in appropriate litigation?