{"id":"AIBD-0985","title":"A Kansas federal court found that Spirit AeroSystems' Predict review, stopped at 85% recall, was reasonable and denied a demand for more documents","summary":"In April 2020 a US magistrate judge found that Spirit AeroSystems' Predict review of 322,000 documents, stopped at 85% recall, was reasonable and that more was not proportional.","date_observed":"2020-04-09","date_basis":"source publication","date_precision":"day","benefit_status":"reported","source_collection":null,"source_entry":null,"account":"Spirit AeroSystems' former chief executive sued the company over pay withheld under his retirement agreement. Months of search term talks over Spirit's electronic files yielded low responsiveness rates. In September 2019 Spirit agreed to a TAR review of the 322,000 documents that the plaintiff's terms and custodians had returned. Its vendor Legility used Predict, a tool that learns continuously from reviewers' coding and ranks documents by likely responsiveness. Spirit stopped after reaching 85% recall, meaning that the algorithm had identified 85% of the responsive documents in the set. Spirit reported that only 3.3% of the set was responsive. It produced or withheld as privileged 23,951 documents, 9,128 of them responsive. On 9 April 2020 Magistrate Judge Angel D. Mitchell found that the review was reasonable. She held that making Spirit review and produce about 800 more documents marked responsive was not proportional. She wrote that the plaintiff had likely received far more documents on the issue than any other form of review would have produced. Spirit's TAR costs came to about $600,000. The court did not test the tool.","category":"benefit created","unexpected":false,"purpose":"A developer that the orders do not name built Predict to rank documents by likely responsiveness, learning continuously from the coding of reviewers.","system":"Predict, a technology-assisted review tool that uses continuous active learning","model_version":null,"developer":"Not named in the orders, which say that Spirit's e-discovery vendor Legility offers the tool","deployer":"Spirit AeroSystems, Inc. (defendant), with its e-discovery vendor Legility and its law firm","location":"Kansas, United States","domain":"Legal services / e-discovery","segment":null,"benefit_type":"Improved Task Accuracy","work_activities":["4.A.2.a.2","4.A.1.a.1"],"taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"organizational","depth":1,"reason":"Spirit reported that its review identified 85% of the responsive documents, and the court held that the review met its discovery obligations."}],"ai_role":{"status":"unassessed","statement":"The court accepted Spirit's 85% recall figure and did not test the tool or compare it with another method. Its view that other review would have produced fewer documents rests on earlier cases. The orders describe a tool that learns continuously from the coding of reviewers."},"evidence":[{"kind":"court document","citation":"United States District Court for the District of Kansas (Magistrate Judge Angel D. Mitchell). Lawson v. Spirit AeroSystems, Inc., No. 18-1100-EFM-ADM, Memorandum and Order (Doc. 335), 9 April 2020. CourtListener RECAP copy of the filed order.","title":"Lawson v. Spirit AeroSystems, Inc., No. 18-1100-EFM-ADM, Memorandum and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.335.0.pdf","date":"2020-04-09","written_by":"independent","quote":"In this case, the court finds that Spirit's TAR was reasonable and production of the residual TAR documents is not proportional to the needs of the case."}],"replication":{"produced_by":["Spirit ran a TAR review of the 322,000 documents that the plaintiff's search terms and custodians had returned.","Predict ranked documents by likely responsiveness and learned continuously from the coding of reviewers.","Legility's contract attorneys did a first-level review, and Spirit's law firm did a second-level review."],"must_hold":["Legility validates the results of the review statistically.","The court judges recall against the facts of the case, with 75% to 85% typical in recent guidance.","Further review must be proportional to the needs of the case."],"echoes":[]},"paired_incident":null,"headline":"Court accepts Spirit's *Predict* review stopped at 85% recall","metric":"85% recall","comparator":{"against":"Any other form of review","before":null,"after":null,"basis":"assumed","quote":"already likely received far more documents bearing on the issue than he would have had Spirit used any other form of review.","url":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.335.0.pdf"},"later":{"status":"litigated","date":"2020-06-18","note":"On 18 June 2020 the court granted Spirit's motion to shift its TAR expenses, reported at about $600,000, to the plaintiff. It found that review of a largely non-responsive data set had become disproportionate to the needs of the case.","quote":"Thus, the ESI/TAR process became disproportionate to the needs of the case.","url":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.372.0.pdf"},"value":null,"developer_country":null,"deployer_country":"United States","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the District of Kansas.","source":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.335.0.pdf","editor":"AIBD"}],"confidence":"found","shade":"high","claim_status":"Found by a court","claim_basis":"Sources: one independent court document. The cited evidence includes no independent study or report.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0985","catalog_version":"1.0.0","source_fingerprint":"62b072484121654b8e65771f08b94d4b30f6814d39815b29be74b3fd29c0c948","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-0985-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Not named in the orders, which say that Spirit's e-discovery vendor Legility offers the tool","deployer":"Spirit AeroSystems, Inc. (defendant), with its e-discovery vendor Legility and its law firm"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"benefit created","ai_role":{"status":"unassessed","statement":"The court accepted Spirit's 85% recall figure and did not test the tool or compare it with another method. Its view that other review would have produced fewer documents rests on earlier cases. The orders describe a tool that learns continuously from the coding of reviewers."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"Spirit reported that its review identified 85% of the responsive documents, and the court held that the review met its discovery obligations."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"found","claim_status":"Found by a court","basis":"Sources: one independent court document. The cited evidence includes no independent study or report.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"Spirit reported that its review identified 85% of the responsive documents, and the court held that the review met its discovery obligations."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"Spirit reported that its review identified 85% of the responsive documents, and the court held that the review met its discovery obligations."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Found by a court. Sources: one independent court document. The cited evidence includes no independent study or report.","evidence_word":"found","claim_status":"Found by a court"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0985-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Found by a court. Sources: one independent court document. The cited evidence includes no independent study or report.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"routine use","answer":"The system was in routine use. Spirit had finished the review and produced the documents when the court ruled.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0985-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was in routine use. Spirit had finished the review and produced the documents when the court ruled.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"later":{"status":"litigated","date":"2020-06-18","note":"On 18 June 2020 the court granted Spirit's motion to shift its TAR expenses, reported at about $600,000, to the plaintiff. It found that review of a largely non-responsive data set had become disproportionate to the needs of the case.","quote":"Thus, the ESI/TAR process became disproportionate to the needs of the case.","url":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.372.0.pdf"}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["later"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0985"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Spirit AeroSystems' former chief executive sued the company over pay withheld under his retirement agreement. Months of search term talks over Spirit's electronic files yielded low responsiveness rates. In September 2019 Spirit agreed to a TAR review of the 322,000 documents that the plaintiff's terms and custodians had returned. Its vendor Legility used Predict, a tool that learns continuously from reviewers' coding and ranks documents by likely responsiveness. Spirit stopped after reaching 85% recall, meaning that the algorithm had identified 85% of the responsive documents in the set. Spirit reported that only 3.3% of the set was responsive. It produced or withheld as privileged 23,951 documents, 9,128 of them responsive. On 9 April 2020 Magistrate Judge Angel D. Mitchell found that the review was reasonable. She held that making Spirit review and produce about 800 more documents marked responsive was not proportional. She wrote that the plaintiff had likely received far more documents on the issue than any other form of review would have produced. Spirit's TAR costs came to about $600,000. The court did not test the tool.","benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"Spirit reported that its review identified 85% of the responsive documents, and the court held that the review met its discovery obligations."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"reported"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"A developer that the orders do not name built Predict to rank documents by likely responsiveness, learning continuously from the coding of reviewers."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"Kansas, United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"A developer that the orders do not name built Predict to rank documents by likely responsiveness, learning continuously from the coding of reviewers."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2020-04-09","date_basis":"source publication","date_precision":"day","revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the District of Kansas.","source":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.335.0.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["Spirit ran a TAR review of the 322,000 documents that the plaintiff's search terms and custodians had returned.","Predict ranked documents by likely responsiveness and learned continuously from the coding of reviewers.","Legility's contract attorneys did a first-level review, and Spirit's law firm did a second-level review."],"must_hold":["Legility validates the results of the review statistically.","The court judges recall against the facts of the case, with 75% to 85% typical in recent guidance.","Further review must be proportional to the needs of the case."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-11-01","change":"Added the record from the finding of the United States District Court for the District of Kansas.","source":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.335.0.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Legal services / e-discovery"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"court document","citation":"United States District Court for the District of Kansas (Magistrate Judge Angel D. Mitchell). Lawson v. Spirit AeroSystems, Inc., No. 18-1100-EFM-ADM, Memorandum and Order (Doc. 335), 9 April 2020. CourtListener RECAP copy of the filed order.","title":"Lawson v. Spirit AeroSystems, Inc., No. 18-1100-EFM-ADM, Memorandum and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.335.0.pdf","date":"2020-04-09","written_by":"independent","quote":"In this case, the court finds that Spirit's TAR was reasonable and production of the residual TAR documents is not proportional to the needs of the case."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0985-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"court document","citation":"United States District Court for the District of Kansas (Magistrate Judge Angel D. Mitchell). Lawson v. Spirit AeroSystems, Inc., No. 18-1100-EFM-ADM, Memorandum and Order (Doc. 335), 9 April 2020. CourtListener RECAP copy of the filed order.","title":"Lawson v. Spirit AeroSystems, Inc., No. 18-1100-EFM-ADM, Memorandum and Order","url":"https://storage.courtlistener.com/recap/gov.uscourts.ksd.120882/gov.uscourts.ksd.120882.335.0.pdf","date":"2020-04-09","written_by":"independent","quote":"In this case, the court finds that Spirit's TAR was reasonable and production of the residual TAR documents is not proportional to the needs of the case."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In April 2020 a US magistrate judge found that Spirit AeroSystems' Predict review of 322,000 documents, stopped at 85% recall, was reasonable and that more was not proportional.","account":"Spirit AeroSystems' former chief executive sued the company over pay withheld under his retirement agreement. Months of search term talks over Spirit's electronic files yielded low responsiveness rates. In September 2019 Spirit agreed to a TAR review of the 322,000 documents that the plaintiff's terms and custodians had returned. Its vendor Legility used Predict, a tool that learns continuously from reviewers' coding and ranks documents by likely responsiveness. Spirit stopped after reaching 85% recall, meaning that the algorithm had identified 85% of the responsive documents in the set. Spirit reported that only 3.3% of the set was responsive. It produced or withheld as privileged 23,951 documents, 9,128 of them responsive. On 9 April 2020 Magistrate Judge Angel D. Mitchell found that the review was reasonable. She held that making Spirit review and produce about 800 more documents marked responsive was not proportional. She wrote that the plaintiff had likely received far more documents on the issue than any other form of review would have produced. Spirit's TAR costs came to about $600,000. The court did not test the tool."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Predict, a technology-assisted review tool that uses continuous active learning"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"A Kansas federal court found that Spirit AeroSystems' Predict review, stopped at 85% recall, was reasonable and denied a demand for more documents","headline":"Court accepts Spirit's *Predict* review stopped at 85% recall"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["92275688e556eb64c4eaa76b8a75e0713dbb85b76acff8c48882118edc8f014d"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Spirit AeroSystems' former chief executive sued the company over pay withheld under his retirement agreement. Months of search term talks over Spirit's electronic files yielded low responsiveness rates. In September 2019 Spirit agreed to a TAR review of the 322,000 documents that the plaintiff's terms and custodians had returned. Its vendor Legility used Predict, a tool that learns continuously from reviewers' coding and ranks documents by likely responsiveness. Spirit stopped after reaching 85% recall, meaning that the algorithm had identified 85% of the responsive documents in the set. Spirit reported that only 3.3% of the set was responsive. It produced or withheld as privileged 23,951 documents, 9,128 of them responsive. On 9 April 2020 Magistrate Judge Angel D. Mitchell found that the review was reasonable. She held that making Spirit review and produce about 800 more documents marked responsive was not proportional. She wrote that the plaintiff had likely received far more documents on the issue than any other form of review would have produced. Spirit's TAR costs came to about $600,000. The court did not test the tool.","replication":{"produced_by":["Spirit ran a TAR review of the 322,000 documents that the plaintiff's search terms and custodians had returned.","Predict ranked documents by likely responsiveness and learned continuously from the coding of reviewers.","Legility's contract attorneys did a first-level review, and Spirit's law firm did a second-level review."],"must_hold":["Legility validates the results of the review statistically.","The court judges recall against the facts of the case, with 75% to 85% typical in recent guidance.","Further review must be proportional to the needs of the case."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":29,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":27,"review_not_assessed":62,"not_assessed":0,"complete":false,"by_state":{"value_present":29,"not_reported":0,"unresolved":62,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"f57de235df5ea27c7c67bd57d584da8e3a22c750bf69465b42cf68072c064f58"}