What happened
Plaintiffs in a 2016 civil case against the City of Chicago disputed how the City would search its emails. A vendor ran the plaintiffs' search terms, which left about 192,000 unique emails, or 1.3 million pages. The City planned to review them with Relativity's Active Learning, which reorders the review queue as attorneys code documents. Attorneys make every coding decision, and the City planned elusion testing, a cut-off score and other quality controls. The plaintiffs asked for refined search terms and a privilege review, or else TAR over the whole collection under their own protocol. On 3 September 2020 Magistrate Judge Young B. Kim denied their motion. He found that culling before TAR satisfied the reasonable inquiry standard and was proportional to the case. He held that Active Learning's quality controls answered the concern that reviewers might train the tool badly. He noted that both sides agreed that TAR is generally far more accurate than manual review or keyword searches. The review had not run, and the court tested neither the tool nor its accuracy.
Sources
-
Court document, by authors with no tie to the system.
United States District Court for the Northern District of Illinois (Magistrate Judge Young B. Kim). Livingston v. City of Chicago, No. 16 CV 10156, Memorandum Opinion and Order (Doc. 309), 3 September 2020. CourtListener RECAP copy of the filed order.
While the court does not discount the possibility that using TAR at the onset might reveal more responsive documents overall, based on the number of documents that were discarded using Plaintiffs' proposed search terms, pre-TAR culling will achieve the best possible review in this case.
Details
AI system
- Workflow
-
A vendor exported the City's emails and ran the plaintiffs' search terms, which left about 192,000 unique emails. Active Learning reorders the review queue every 20 minutes from the attorneys' coding decisions. The City planned elusion testing, graphing, family reconciliation and a cut-off score to check its review.
Follow-up
- Conditions for replication
-
Search terms remove most of a collection in which few documents are responsive. The City discloses the software that it uses and how it will validate the results. Reviewers sample the unreviewed documents before they stop.
Entry details
- Sector
-
Legal services / e-discovery.
- Purpose
-
Relativity built Active Learning to reorder a review queue from the coding decisions of attorneys, so that the documents most likely to be responsive reach reviewers first.
31 more questions not yet checked
model, release, provider, role, configuration, inputs and outputs, autonomy, human oversight, AI contribution, beneficiaries, reach, distribution and access, duration, every measurement question, every money question, ownership and control, dependencies, cross-border connections, geopolitical implications, replication attempts, availability, institutional decision, grounds, action and result, success criterion, every evidence question, observed or intended, alternative assessments, related entries, other identifiers, category mappings.
Cite this record
Peter H. Vartanian. (2026). Court backs Chicago's Active Learning review of 1.3 million pages (AIBD-0984). Artificial Intelligence Benefits Database, edition of 1 November 2026. https://petervartanian.xyz/aibd/records/AIBD-0984.html (archived at https://doi.org/10.7910/DVN/AADSII, version 7.0)