What happened
Three residents challenged New York City's community preference policy for affordable housing. The court had directed the City to move from manual review to TAR to speed up its document production. The plaintiffs argued that reviewers had marked responsive documents non-responsive and so trained the software badly. Magistrate Judge Katharine H. Parker examined the City's training and validation in camera. The seed set contained over 7,200 documents, and the City ran five training rounds and then a validation process. On 27 November 2017 she found that the City had trained and used its TAR system appropriately. The plaintiffs identified 5 documents wrongly marked non-responsive out of the 100,000 reviewed, and she attributed such errors to human reviewers. She held that this did not put the accuracy and reliability of the TAR process as a whole in question. She ordered the City to run the plaintiffs' added search terms through its existing TAR process. She also ordered the City to give the plaintiffs random samples of documents marked non-responsive. The opinion reports no recall figure, and the City's submissions stayed confidential.
Sources
-
Court document, by authors with no tie to the system.
United States District Court for the Southern District of New York (Magistrate Judge Katharine H. Parker). Winfield v. City of New York, No. 15-CV-05236 (LTS) (KHP), Opinion and Order (Doc. 217), 27 November 2017. CourtListener RECAP copy of the filed opinion.
These in camera submissions reveal that the City appropriately trained and utilized its TAR system. The City's seed set included over 7,200 documents that were reviewed by the City's document review team and marked as responsive or non- responsive in order to train the system.
Details
AI system
- Workflow
-
The City trained its system on a seed set of over 7,200 documents that its review team coded. The seed set mixed random documents with pleadings, the plaintiffs' document requests and other example documents. The City ran five training rounds and then a validation process, and trained its reviewers on the issues in the case.
Follow-up
- Conditions for replication
-
Human reviewers code documents in line with the document requests, since their errors can leave gaps in production. The producing party shows its training and validation to the court when the other side challenges them. The requesting party can sample documents marked non-responsive.
Entry details
- Sector
-
Legal services / e-discovery.
- Purpose
-
A vendor that the opinion does not name built the City's TAR software to sort documents into responsive and non-responsive and rank them by relevance, after training on a seed set.
31 more questions not yet checked
model, release, provider, role, configuration, inputs and outputs, autonomy, human oversight, AI contribution, beneficiaries, reach, distribution and access, duration, every measurement question, every money question, ownership and control, dependencies, cross-border connections, geopolitical implications, replication attempts, availability, institutional decision, grounds, action and result, success criterion, every evidence question, observed or intended, alternative assessments, related entries, other identifiers, category mappings.
Cite this record
Peter H. Vartanian. (2026). Judge clears New York City's TAR system after in camera review (AIBD-0986). Artificial Intelligence Benefits Database, edition of 1 November 2026. https://petervartanian.xyz/aibd/records/AIBD-0986.html (archived at https://doi.org/10.7910/DVN/AADSII, version 7.0)