{"id":"AIBD-0986","title":"A New York federal court examined New York City's TAR training in camera and found that the City had trained and used the system appropriately","summary":"In November 2017 a US magistrate judge examined New York City's TAR training in camera. She found that the City had trained and used the system appropriately and that its process was reliable.","date_observed":"2017-11-27","date_basis":"source publication","date_precision":"day","benefit_status":"reported","source_collection":null,"source_entry":null,"account":"Three residents challenged New York City's community preference policy for affordable housing. The court had directed the City to move from manual review to TAR to speed up its document production. The plaintiffs argued that reviewers had marked responsive documents non-responsive and so trained the software badly. Magistrate Judge Katharine H. Parker examined the City's training and validation in camera. The seed set contained over 7,200 documents, and the City ran five training rounds and then a validation process. On 27 November 2017 she found that the City had trained and used its TAR system appropriately. The plaintiffs identified 5 documents wrongly marked non-responsive out of the 100,000 reviewed, and she attributed such errors to human reviewers. She held that this did not put the accuracy and reliability of the TAR process as a whole in question. She ordered the City to run the plaintiffs' added search terms through its existing TAR process. She also ordered the City to give the plaintiffs random samples of documents marked non-responsive. The opinion reports no recall figure, and the City's submissions stayed confidential.","category":"benefit created","unexpected":false,"purpose":"A vendor that the opinion does not name built the City's TAR software to sort documents into responsive and non-responsive and rank them by relevance, after training on a seed set.","system":"The City of New York's technology assisted review (predictive coding) software, not named in the opinion","model_version":null,"developer":"Not named in the opinion","deployer":"City of New York (defendant) and its document review team","location":"New York, United States","domain":"Legal services / e-discovery","segment":null,"benefit_type":"Improved Task Accuracy","work_activities":["4.A.2.a.2","4.A.1.a.1"],"taxonomy_note":"The court found the process reliable. Its statement that TAR is more accurate than manual review rests on earlier cases.","registry_ids":{},"ratings":[{"group":"organizational","depth":1,"reason":"The court found that the City's TAR review was reliable and let it keep using the system for its remaining review."}],"ai_role":{"status":"unassessed","statement":"The court examined the City's training and validation in camera but reports no recall or precision figure. It did not compare the TAR review with manual review. The system learned from a seed set that the City's reviewers coded."},"evidence":[{"kind":"court document","citation":"United States District Court for the Southern District of New York (Magistrate Judge Katharine H. Parker). Winfield v. City of New York, No. 15-CV-05236 (LTS) (KHP), Opinion and Order (Doc. 217), 27 November 2017. CourtListener RECAP copy of the filed opinion.","title":"Winfield v. City of New York, No. 15-CV-05236 (LTS) (KHP), Opinion and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.nysd.444418/gov.uscourts.nysd.444418.217.0.pdf","date":"2017-11-27","written_by":"independent","quote":"These in camera submissions reveal that the City appropriately trained and utilized its TAR system. The City's seed set included over 7,200 documents that were reviewed by the City's document review team and marked as responsive or non- responsive in order to train the system."}],"replication":{"produced_by":["The City trained its system on a seed set of over 7,200 documents that its review team coded.","The seed set mixed random documents with pleadings, the plaintiffs' document requests and other example documents.","The City ran five training rounds and then a validation process, and trained its reviewers on the issues in the case."],"must_hold":["Human reviewers code documents in line with the document requests, since their errors can leave gaps in production.","The producing party shows its training and validation to the court when the other side challenges them.","The requesting party can sample documents marked non-responsive."],"echoes":[]},"paired_incident":null,"headline":"Judge clears New York City's *TAR* system after in camera review","metric":"5 documents wrongly marked non-responsive out of the 100,000 reviewed","comparator":{"against":"Manual review","before":null,"after":null,"basis":"assumed","quote":"TAR allows parties to prioritize and/or categorize documents for purposes of document review and has been shown to produce more accurate results than manual review.","url":"https://storage.courtlistener.com/recap/gov.uscourts.nysd.444418/gov.uscourts.nysd.444418.217.0.pdf"},"later":null,"value":null,"developer_country":null,"deployer_country":"United States","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the Southern District of New York.","source":"https://storage.courtlistener.com/recap/gov.uscourts.nysd.444418/gov.uscourts.nysd.444418.217.0.pdf","editor":"AIBD"}],"confidence":"found","shade":"high","claim_status":"Found by a court","claim_basis":"Sources: one independent court document. The cited evidence includes no independent study or report.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0986","catalog_version":"1.0.0","source_fingerprint":"956b2a089ba32e977358522c50951adc7c7c0b6584bae74ebd7f26cdabc22650","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-0986-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Not named in the opinion","deployer":"City of New York (defendant) and its document review team"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"benefit created","ai_role":{"status":"unassessed","statement":"The court examined the City's training and validation in camera but reports no recall or precision figure. It did not compare the TAR review with manual review. The system learned from a seed set that the City's reviewers coded."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"The court found that the City's TAR review was reliable and let it keep using the system for its remaining review."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"found","claim_status":"Found by a court","basis":"Sources: one independent court document. The cited evidence includes no independent study or report.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"The court found that the City's TAR review was reliable and let it keep using the system for its remaining review."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"The court found that the City's TAR review was reliable and let it keep using the system for its remaining review."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Found by a court. Sources: one independent court document. The cited evidence includes no independent study or report.","evidence_word":"found","claim_status":"Found by a court"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0986-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Found by a court. Sources: one independent court document. The cited evidence includes no independent study or report.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"routine use","answer":"The system was in routine use. The City was using the system in its document review when the court ruled.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0986-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was in routine use. The City was using the system in its document review when the court ruled.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0986"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Task Accuracy","taxonomy_note":"The court found the process reliable. Its statement that TAR is more accurate than manual review rests on earlier cases."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type","taxonomy_note"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Three residents challenged New York City's community preference policy for affordable housing. The court had directed the City to move from manual review to TAR to speed up its document production. The plaintiffs argued that reviewers had marked responsive documents non-responsive and so trained the software badly. Magistrate Judge Katharine H. Parker examined the City's training and validation in camera. The seed set contained over 7,200 documents, and the City ran five training rounds and then a validation process. On 27 November 2017 she found that the City had trained and used its TAR system appropriately. The plaintiffs identified 5 documents wrongly marked non-responsive out of the 100,000 reviewed, and she attributed such errors to human reviewers. She held that this did not put the accuracy and reliability of the TAR process as a whole in question. She ordered the City to run the plaintiffs' added search terms through its existing TAR process. She also ordered the City to give the plaintiffs random samples of documents marked non-responsive. The opinion reports no recall figure, and the City's submissions stayed confidential.","benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"The court found that the City's TAR review was reliable and let it keep using the system for its remaining review."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"reported"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"A vendor that the opinion does not name built the City's TAR software to sort documents into responsive and non-responsive and rank them by relevance, after training on a seed set."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"New York, United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"A vendor that the opinion does not name built the City's TAR software to sort documents into responsive and non-responsive and rank them by relevance, after training on a seed set."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2017-11-27","date_basis":"source publication","date_precision":"day","revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the Southern District of New York.","source":"https://storage.courtlistener.com/recap/gov.uscourts.nysd.444418/gov.uscourts.nysd.444418.217.0.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["The City trained its system on a seed set of over 7,200 documents that its review team coded.","The seed set mixed random documents with pleadings, the plaintiffs' document requests and other example documents.","The City ran five training rounds and then a validation process, and trained its reviewers on the issues in the case."],"must_hold":["Human reviewers code documents in line with the document requests, since their errors can leave gaps in production.","The producing party shows its training and validation to the court when the other side challenges them.","The requesting party can sample documents marked non-responsive."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the Southern District of New York.","source":"https://storage.courtlistener.com/recap/gov.uscourts.nysd.444418/gov.uscourts.nysd.444418.217.0.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Legal services / e-discovery"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"court document","citation":"United States District Court for the Southern District of New York (Magistrate Judge Katharine H. Parker). Winfield v. City of New York, No. 15-CV-05236 (LTS) (KHP), Opinion and Order (Doc. 217), 27 November 2017. CourtListener RECAP copy of the filed opinion.","title":"Winfield v. City of New York, No. 15-CV-05236 (LTS) (KHP), Opinion and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.nysd.444418/gov.uscourts.nysd.444418.217.0.pdf","date":"2017-11-27","written_by":"independent","quote":"These in camera submissions reveal that the City appropriately trained and utilized its TAR system. The City's seed set included over 7,200 documents that were reviewed by the City's document review team and marked as responsive or non- responsive in order to train the system."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0986-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"court document","citation":"United States District Court for the Southern District of New York (Magistrate Judge Katharine H. Parker). Winfield v. City of New York, No. 15-CV-05236 (LTS) (KHP), Opinion and Order (Doc. 217), 27 November 2017. CourtListener RECAP copy of the filed opinion.","title":"Winfield v. City of New York, No. 15-CV-05236 (LTS) (KHP), Opinion and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.nysd.444418/gov.uscourts.nysd.444418.217.0.pdf","date":"2017-11-27","written_by":"independent","quote":"These in camera submissions reveal that the City appropriately trained and utilized its TAR system. The City's seed set included over 7,200 documents that were reviewed by the City's document review team and marked as responsive or non- responsive in order to train the system."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In November 2017 a US magistrate judge examined New York City's TAR training in camera. She found that the City had trained and used the system appropriately and that its process was reliable.","account":"Three residents challenged New York City's community preference policy for affordable housing. The court had directed the City to move from manual review to TAR to speed up its document production. The plaintiffs argued that reviewers had marked responsive documents non-responsive and so trained the software badly. Magistrate Judge Katharine H. Parker examined the City's training and validation in camera. The seed set contained over 7,200 documents, and the City ran five training rounds and then a validation process. On 27 November 2017 she found that the City had trained and used its TAR system appropriately. The plaintiffs identified 5 documents wrongly marked non-responsive out of the 100,000 reviewed, and she attributed such errors to human reviewers. She held that this did not put the accuracy and reliability of the TAR process as a whole in question. She ordered the City to run the plaintiffs' added search terms through its existing TAR process. She also ordered the City to give the plaintiffs random samples of documents marked non-responsive. The opinion reports no recall figure, and the City's submissions stayed confidential."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"The City of New York's technology assisted review (predictive coding) software, not named in the opinion"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"A New York federal court examined New York City's TAR training in camera and found that the City had trained and used the system appropriately","headline":"Judge clears New York City's *TAR* system after in camera review"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["e0792a91b3a4a3c73dae9d96748082891864a74af5730a80268584698273148e"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Three residents challenged New York City's community preference policy for affordable housing. The court had directed the City to move from manual review to TAR to speed up its document production. The plaintiffs argued that reviewers had marked responsive documents non-responsive and so trained the software badly. Magistrate Judge Katharine H. Parker examined the City's training and validation in camera. The seed set contained over 7,200 documents, and the City ran five training rounds and then a validation process. On 27 November 2017 she found that the City had trained and used its TAR system appropriately. The plaintiffs identified 5 documents wrongly marked non-responsive out of the 100,000 reviewed, and she attributed such errors to human reviewers. She held that this did not put the accuracy and reliability of the TAR process as a whole in question. She ordered the City to run the plaintiffs' added search terms through its existing TAR process. She also ordered the City to give the plaintiffs random samples of documents marked non-responsive. The opinion reports no recall figure, and the City's submissions stayed confidential.","replication":{"produced_by":["The City trained its system on a seed set of over 7,200 documents that its review team coded.","The seed set mixed random documents with pleadings, the plaintiffs' document requests and other example documents.","The City ran five training rounds and then a validation process, and trained its reviewers on the issues in the case."],"must_hold":["Human reviewers code documents in line with the document requests, since their errors can leave gaps in production.","The producing party shows its training and validation to the court when the other side challenges them.","The requesting party can sample documents marked non-responsive."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":28,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":26,"review_not_assessed":63,"not_assessed":0,"complete":false,"by_state":{"value_present":28,"not_reported":0,"unresolved":63,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"a9e9846f6a77ee5fa8f94673b380bcd1dc0099da035c6fda615ff795d12e41ef"}