Court Resolves Disputes Over Number of Custodians and Validation Protocol

In Tremblay v. OpenAI, Inc., 2024 WL 3638421 (N.D. Cal. July 31, 2024), the court resolved two disputes over an ESI Protocol, writing:

• “As to the first category of disagreement, the Parties have found themselves at an impasse as to the proper number of document custodians to be set forth in the ESI protocol.”
• “The second category of disagreement pertains to the Parties dispute about an appropriate validation protocol for ESI review and discovery.” That category contained several subcategories.

DISPUTE OVER THE NUMBER OF CUSTODIANS

Plaintiffs identified 43 potential defense custodians and offered to compromise by requesting only 28 custodians.  Plaintiffs sought to reserve the right to “to seek more th[o]rough agreement or by seeking further court intervention in the future.”

In my words, plaintiffs essentially suggested phased discovery with an offered agreement on phase 1 and both parties reserving all rights if phase 2 arose.

However, according to plaintiffs, defendants would agree to only a range of 18 – 25 custodians.  And, there was “an impasse on the procedure for identifying custodians, and whether the requesting Party should be capped at 5 additional custodians if it determines more are necessary.”  The nature of the identification impasse was not described.

Plaintiffs reported that they had offered to accept the 5-person cap, “if Defendants agreed to resolve the total initial count somewhere in the range of 23-25 persons, but Defendants reportedly rejected this proposal.”

The court wrote that: “Defendants propose an initial floor of 19 custodians and claim that their own search indicates that there might be only 12 custodians that are ‘most likely to have unique discoverable information responsive to Plaintiffs’ requests.’”

Defendants were offering seven more custodians than they thought were responsive.  Plaintiffs offered to accept fifteen less than they considered potentially responsive.

Thus, the dispute seems to be over the 19 custodians proposed by defendants versus the 23 – 25 custodians suggested by plaintiffs and the method of identifying them, with an agreed 5-person cap on additional custodians.

Sometimes, it may “be in the parties’ interest to reach a negotiated result. It should not be forgotten that a party could be ‘right,’ but find itself on the losing side of a court’s ruling.”  See There’s No Right or Wrong Answer – But There Are Mistakes (Apr. 19, 2024)(citation omitted). To the same effect, it “may behoove” parties to “reach an accord” between themselves, “because courts have great discretion to resolve discovery disputes.”  See What Happens When Parties Fail to Negotiate a Privilege Log in Good Faith? (Apr. 8, 2024)(citation omitted).

Plaintiff appears to have substantially prevailed.  The court ruled that:

The court will set the initial number of custodians at 24, however, to the extent that Plaintiffs can identify any number of additional custodians (not limited to 5) that Plaintiffs can persuasively argue to possess non-cumulative, unique documents or information that would justify enlarging the list of custodians, the undersigned will hear those arguments through future filings of the Parties using the joint letter-brief process. As to the identification of appropriate custodians with non-duplicative documents or information, the Parties are ORDERED to work together in good-faith to make those identifications without the need for further court intervention. If, however, a discreet dispute arises in that context after good-faith and genuine efforts to resolve or narrow the dispute, the Parties may present it through the joint letter-brief process. [emphasis added].

DISPUTES OVER APPROPRIATE VALIDATION

Turning to validation, the court wrote: “Specifically, the Parties’ disagreement in this respect consists of disputes as to: (1) whether the producing Party should disclose its search terms before the review process (and whether input from the requesting party should be considered in determining search terms); (2) whether recall or elusion is the appropriate evaluation method for validating the efficacy of a search; and (3) whether Plaintiffs’ proposal of 98% +/-2% or Defendants’ proposal of 95% +/-5% is the appropriate confidence metric for sampling.”

The proposed confidence levels differed by 3% and the proposed margins of error differed by 3%.  Plaintiffs proposal of 98% +/-2% would require that defendants review more documents than would otherwise be reviewed under defendants’ proposal of 95% +/-5%.  In short, to achieve a higher confidence level, more effort is required.  The number of additional documents would be dependent on the size of the document corpus and was not provided in the court’s opinion.

“Recall” and “elusion” are terms of art.

“Recall,” by definition, is “’an information retrieval performance measure that quantifies the fraction of known relevant documents which were effectively retrieved.’ That is, out of the total number of relevant documents in the document collection, how many were retrieved correctly?” See “The Sedona Conference Commentary on Search and Retrieval,” 15 Sedona Conf. J. at 237 (2014).  Recall is the number of responsive documents retrieved by a search, divided by the number of responsive documents in the dataset as a whole. Id. at 238. Recall generally is analyzed together with “precision.” Id. at 238.

“Elusion” testing is a standard validation measure in which a sample from the null set, or discard  pile, is reviewed.  “[T]his type of validation measure is a standard quality assurance practice in electronic discovery and essential for providing transparency on whether material documents have eluded the search terms.”  Deal Genius, LLC v. O2COOL, LLC, 2023 WL 3445758, at *3 (N.D. Ill. Apr. 28, 2023)(Special Master Phil Favro).

The Sedona Conference defines the term “elusion” as “[t]he percentage of documents of a search’s null set that were missed by the search, usually determined with review of a random sample of the null set. The elusion rate can be multiplied by the number of documents in the null set to estimate how many documents were missed by the search.”

Deal Genius, LLC v. O2COOL, LLC, 682 F. Supp. 3d 727, 730 n. 7 (N.D. Ill. Jul. 14, 2023).

As to search term disclosure, the court ruled that “the disclosure of search terms is appropriate and reasonable – thus, to that extent, Plaintiffs’ request is GRAN[T]ED. However, Plaintiffs’ suggestion that the requesting Party have input in determining [the opposing party’s] search terms raises the specter of too many future delays and disputes over methodology and search term formulation. Accordingly, to that extent, Plaintiffs’ request is DENIED.”  The court then ordered specific protocol language.

It next turned to metrics.  In my words, it ordered sunshine and cooperation, but did not order details:  “As to the Parties’ remaining disputes about validation, Plaintiffs’ request is GRANTED in part, and the court directs the Parties to disclose their evaluation method for validating the efficiency of searches as well as their method for identifying the confidence metric for their sampling.”  [emphasis added].  It ordered the following protocol language:

Each Producing Party shall take reasonable steps to validate its review process (i.e., using quality control measures to determine whether its production is missing relevant ESI or contains substantial amounts of irrelevant ESI) and make any necessary adjustments or corrections to its process. If, after reviewing a Producing Party’s production, a Requesting Party reasonably requests additional information regarding the validation method(s) used by the Producing Party, the Producing Party will disclose the level of end-to-end recall (the percentage of responsive Documents in the collection which were identified as responsive by that Producing Party’s methodology). If there remain disputes between the Parties regarding validation, the Parties shall meet and confer to resolve such disputes in good faith, including a reasonable discussion regarding the tool used and the parameters used to obtain or calculate the level of recall.

“ESI Protocols” are not mentioned in the Federal Rules of Civil Procedure.”  “ESI Protocol” v. “Discovery Plan”   (Jan. 2, 2024).  Because the term “ESI Protocol” is not rules-based, it is, as Lewis Carroll wrote, “what I choose it to mean — neither more nor less….” Id.