{"id":"AIBD-0984","title":"A Chicago federal court found that the City's plan to review 1.3 million pages with Relativity Active Learning was reasonable and proportional","summary":"In September 2020 a US magistrate judge found that Chicago's plan to review 1.3 million pages with Relativity Active Learning was reasonable and proportional to the case.","date_observed":"2020-09-03","date_basis":"source publication","date_precision":"day","benefit_status":"intended","source_collection":null,"source_entry":null,"account":"Plaintiffs in a 2016 civil case against the City of Chicago disputed how the City would search its emails. A vendor ran the plaintiffs' search terms, which left about 192,000 unique emails, or 1.3 million pages. The City planned to review them with Relativity's Active Learning, which reorders the review queue as attorneys code documents. Attorneys make every coding decision, and the City planned elusion testing, a cut-off score and other quality controls. The plaintiffs asked for refined search terms and a privilege review, or else TAR over the whole collection under their own protocol. On 3 September 2020 Magistrate Judge Young B. Kim denied their motion. He found that culling before TAR satisfied the reasonable inquiry standard and was proportional to the case. He held that Active Learning's quality controls answered the concern that reviewers might train the tool badly. He noted that both sides agreed that TAR is generally far more accurate than manual review or keyword searches. The review had not run, and the court tested neither the tool nor its accuracy.","category":"benefit created","unexpected":false,"purpose":"Relativity built Active Learning to reorder a review queue from the coding decisions of attorneys, so that the documents most likely to be responsive reach reviewers first.","system":"Relativity Active Learning (AL)","model_version":null,"developer":"Relativity (Relativity ODA LLC)","deployer":"City of Chicago (defendant) and its attorneys","location":"Chicago, Illinois, United States","domain":"Legal services / e-discovery","segment":null,"benefit_type":"Improved Task Accuracy","work_activities":["4.A.2.b.6","4.A.2.a.2"],"taxonomy_note":"The court found the plan reasonable and proportional. The accuracy claim is the view that both parties shared, which the court noted.","registry_ids":{},"ratings":[{"group":"organizational","depth":null,"reason":"The review had not run when the court ruled, and the order states no measured accuracy, time or cost."}],"ai_role":{"status":"unassessed","statement":"The court approved the plan before the review ran and did not test the tool. The accuracy claim is the parties' shared view. The order describes software that uses learning algorithms to reorder documents from the attorneys' coding decisions."},"evidence":[{"kind":"court document","citation":"United States District Court for the Northern District of Illinois (Magistrate Judge Young B. Kim). Livingston v. City of Chicago, No. 16 CV 10156, Memorandum Opinion and Order (Doc. 309), 3 September 2020. CourtListener RECAP copy of the filed order.","title":"Livingston v. City of Chicago, No. 16 CV 10156, Memorandum Opinion and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.ilnd.332956/gov.uscourts.ilnd.332956.309.0_1.pdf","date":"2020-09-03","written_by":"independent","quote":"While the court does not discount the possibility that using TAR at the onset might reveal more responsive documents overall, based on the number of documents that were discarded using Plaintiffs' proposed search terms, pre-TAR culling will achieve the best possible review in this case."}],"replication":{"produced_by":["A vendor exported the City's emails and ran the plaintiffs' search terms, which left about 192,000 unique emails.","Active Learning reorders the review queue every 20 minutes from the attorneys' coding decisions.","The City planned elusion testing, graphing, family reconciliation and a cut-off score to check its review."],"must_hold":["Search terms remove most of a collection in which few documents are responsive.","The City discloses the software that it uses and how it will validate the results.","Reviewers sample the unreviewed documents before they stop."],"echoes":[]},"paired_incident":null,"headline":"Court backs Chicago's *Active Learning* review of 1.3 million pages","metric":"1.3 million pages","comparator":{"against":"Manual review or keyword searches","before":null,"after":null,"basis":"assumed","quote":"Significantly, the parties agree that generally TAR is a far more accurate means of producing responsive ESI than manual review or keyword searches.","url":"https://storage.courtlistener.com/recap/gov.uscourts.ilnd.332956/gov.uscourts.ilnd.332956.309.0_1.pdf"},"later":null,"value":null,"developer_country":"United States","deployer_country":"United States","unrated_because":"The court approved the City's plan before the review ran, so the order reports no measured result.","brief_notes":null,"brief":null,"revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the Northern District of Illinois.","source":"https://storage.courtlistener.com/recap/gov.uscourts.ilnd.332956/gov.uscourts.ilnd.332956.309.0_1.pdf","editor":"AIBD"}],"confidence":"found","shade":"high","claim_status":"Found by a court","claim_basis":"Sources: one independent court document. The cited evidence includes no independent study or report.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0984","catalog_version":"1.0.0","source_fingerprint":"53e2ef528d47a8baf2403fa35e00e4a4ea76b91191ea304e861fff643aa865e4","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-0984-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Relativity (Relativity ODA LLC)","deployer":"City of Chicago (defendant) and its attorneys"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"benefit created","ai_role":{"status":"unassessed","statement":"The court approved the plan before the review ran and did not test the tool. The accuracy claim is the parties' shared view. The order describes software that uses learning algorithms to reorder documents from the attorneys' coding decisions."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"found","claim_status":"Found by a court","basis":"Sources: one independent court document. The cited evidence includes no independent study or report.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The review had not run when the court ruled, and the order states no measured accuracy, time or cost."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The review had not run when the court ruled, and the order states no measured accuracy, time or cost."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Found by a court. Sources: one independent court document. The cited evidence includes no independent study or report.","evidence_word":"found","claim_status":"Found by a court"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0984-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Found by a court. Sources: one independent court document. The cited evidence includes no independent study or report.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"proposed","answer":"The use was proposed. The court approved the City's plan before the review ran.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0984-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The use was proposed. The court approved the City's plan before the review ran.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0984"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Task Accuracy","taxonomy_note":"The court found the plan reasonable and proportional. The accuracy claim is the view that both parties shared, which the court noted."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type","taxonomy_note"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Plaintiffs in a 2016 civil case against the City of Chicago disputed how the City would search its emails. A vendor ran the plaintiffs' search terms, which left about 192,000 unique emails, or 1.3 million pages. The City planned to review them with Relativity's Active Learning, which reorders the review queue as attorneys code documents. Attorneys make every coding decision, and the City planned elusion testing, a cut-off score and other quality controls. The plaintiffs asked for refined search terms and a privilege review, or else TAR over the whole collection under their own protocol. On 3 September 2020 Magistrate Judge Young B. Kim denied their motion. He found that culling before TAR satisfied the reasonable inquiry standard and was proportional to the case. He held that Active Learning's quality controls answered the concern that reviewers might train the tool badly. He noted that both sides agreed that TAR is generally far more accurate than manual review or keyword searches. The review had not run, and the court tested neither the tool nor its accuracy.","benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The review had not run when the court ruled, and the order states no measured accuracy, time or cost."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"intended"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"Relativity built Active Learning to reorder a review queue from the coding decisions of attorneys, so that the documents most likely to be responsive reach reviewers first."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"Chicago, Illinois, United States","developer_country":"United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"Relativity built Active Learning to reorder a review queue from the coding decisions of attorneys, so that the documents most likely to be responsive reach reviewers first."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2020-09-03","date_basis":"source publication","date_precision":"day","revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the Northern District of Illinois.","source":"https://storage.courtlistener.com/recap/gov.uscourts.ilnd.332956/gov.uscourts.ilnd.332956.309.0_1.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["A vendor exported the City's emails and ran the plaintiffs' search terms, which left about 192,000 unique emails.","Active Learning reorders the review queue every 20 minutes from the attorneys' coding decisions.","The City planned elusion testing, graphing, family reconciliation and a cut-off score to check its review."],"must_hold":["Search terms remove most of a collection in which few documents are responsive.","The City discloses the software that it uses and how it will validate the results.","Reviewers sample the unreviewed documents before they stop."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the Northern District of Illinois.","source":"https://storage.courtlistener.com/recap/gov.uscourts.ilnd.332956/gov.uscourts.ilnd.332956.309.0_1.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Legal services / e-discovery"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"court document","citation":"United States District Court for the Northern District of Illinois (Magistrate Judge Young B. Kim). Livingston v. City of Chicago, No. 16 CV 10156, Memorandum Opinion and Order (Doc. 309), 3 September 2020. CourtListener RECAP copy of the filed order.","title":"Livingston v. City of Chicago, No. 16 CV 10156, Memorandum Opinion and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.ilnd.332956/gov.uscourts.ilnd.332956.309.0_1.pdf","date":"2020-09-03","written_by":"independent","quote":"While the court does not discount the possibility that using TAR at the onset might reveal more responsive documents overall, based on the number of documents that were discarded using Plaintiffs' proposed search terms, pre-TAR culling will achieve the best possible review in this case."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0984-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"court document","citation":"United States District Court for the Northern District of Illinois (Magistrate Judge Young B. Kim). Livingston v. City of Chicago, No. 16 CV 10156, Memorandum Opinion and Order (Doc. 309), 3 September 2020. CourtListener RECAP copy of the filed order.","title":"Livingston v. City of Chicago, No. 16 CV 10156, Memorandum Opinion and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.ilnd.332956/gov.uscourts.ilnd.332956.309.0_1.pdf","date":"2020-09-03","written_by":"independent","quote":"While the court does not discount the possibility that using TAR at the onset might reveal more responsive documents overall, based on the number of documents that were discarded using Plaintiffs' proposed search terms, pre-TAR culling will achieve the best possible review in this case."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In September 2020 a US magistrate judge found that Chicago's plan to review 1.3 million pages with Relativity Active Learning was reasonable and proportional to the case.","account":"Plaintiffs in a 2016 civil case against the City of Chicago disputed how the City would search its emails. A vendor ran the plaintiffs' search terms, which left about 192,000 unique emails, or 1.3 million pages. The City planned to review them with Relativity's Active Learning, which reorders the review queue as attorneys code documents. Attorneys make every coding decision, and the City planned elusion testing, a cut-off score and other quality controls. The plaintiffs asked for refined search terms and a privilege review, or else TAR over the whole collection under their own protocol. On 3 September 2020 Magistrate Judge Young B. Kim denied their motion. He found that culling before TAR satisfied the reasonable inquiry standard and was proportional to the case. He held that Active Learning's quality controls answered the concern that reviewers might train the tool badly. He noted that both sides agreed that TAR is generally far more accurate than manual review or keyword searches. The review had not run, and the court tested neither the tool nor its accuracy."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Relativity Active Learning (AL)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"A Chicago federal court found that the City's plan to review 1.3 million pages with Relativity Active Learning was reasonable and proportional","headline":"Court backs Chicago's *Active Learning* review of 1.3 million pages"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["2484d25aa9151addc6de4e528652840852eb53c33cd0d8f81b0c868ddda30157"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Plaintiffs in a 2016 civil case against the City of Chicago disputed how the City would search its emails. A vendor ran the plaintiffs' search terms, which left about 192,000 unique emails, or 1.3 million pages. The City planned to review them with Relativity's Active Learning, which reorders the review queue as attorneys code documents. Attorneys make every coding decision, and the City planned elusion testing, a cut-off score and other quality controls. The plaintiffs asked for refined search terms and a privilege review, or else TAR over the whole collection under their own protocol. On 3 September 2020 Magistrate Judge Young B. Kim denied their motion. He found that culling before TAR satisfied the reasonable inquiry standard and was proportional to the case. He held that Active Learning's quality controls answered the concern that reviewers might train the tool badly. He noted that both sides agreed that TAR is generally far more accurate than manual review or keyword searches. The review had not run, and the court tested neither the tool nor its accuracy.","replication":{"produced_by":["A vendor exported the City's emails and ran the plaintiffs' search terms, which left about 192,000 unique emails.","Active Learning reorders the review queue every 20 minutes from the attorneys' coding decisions.","The City planned elusion testing, graphing, family reconciliation and a cut-off score to check its review."],"must_hold":["Search terms remove most of a collection in which few documents are responsive.","The City discloses the software that it uses and how it will validate the results.","Reviewers sample the unreviewed documents before they stop."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":27,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":25,"review_not_assessed":64,"not_assessed":0,"complete":false,"by_state":{"value_present":27,"not_reported":0,"unresolved":64,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"fb8c6b185a6f998be61b290e1d4651bc19c09bc14f6223d98e2144e2e21d1424"}