{"id":"AIBD-0834","title":"The US Tax Court held that a predictive coding review of about 406,000 documents gave the IRS a complete response, at a precision of 3%","summary":"In July 2016 the US Tax Court held that a predictive coding response to the IRS was complete. The model sorted about 406,000 documents into a production set of about 180,000, at 3% precision.","date_observed":"2016-07-13","date_basis":"source publication","date_precision":"day","benefit_status":"mixed","source_collection":null,"source_entry":null,"account":"After the Tax Court allowed predictive coding in 2014, the parties agreed on a protocol. The Commissioner coded two random sets of 1,000 documents and twelve smaller sets to train and test the model. He chose a recall rate of 95%. The taxpayers ran the model over about 406,000 documents and ran a second algorithm to find privileged material. In January and March 2016 they delivered a production set of about 180,000 documents. The Commissioner kept 5,796 and returned the rest. He then moved to compel documents that his Boolean search had hit and the model had left out. On 13 July 2016 Judge Ronald L. Buch denied the motion. He held that the inquiry was reasonable and that the response was complete. He assumed for argument that the model's recall fell short. He noted that the 95% recall setting brought a precision of 3%. He called the belief that human review is perfect a myth.","category":"benefit created","unexpected":false,"purpose":"A vendor that the order does not name built the predictive coding software to find responsive documents, with a model trained on the requesting party's coding.","system":"Predictive coding algorithm","model_version":null,"developer":"Not named in the order","deployer":"Dynamo Holdings Limited Partnership and Beekman Vista, Inc. (petitioners)","location":"United States","domain":"Legal services / e-discovery","segment":null,"benefit_type":"Repetitive Task Automation","work_activities":["4.A.2.a.2","4.A.1.a.1"],"taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"organizational","depth":1,"reason":"The taxpayers met the request with a set that the model chose from about 406,000 documents, and the order states no time or cost saved."}],"ai_role":{"status":"unassessed","statement":"The court judged whether the response met the discovery rules and did not compare the model with a full human review. It assumed for argument that the model's recall fell short and noted a precision of 3%. The model learned from seed sets that the Commissioner coded."},"evidence":[{"kind":"court document","citation":"United States Tax Court (Judge Ronald L. Buch). Dynamo Holdings Limited Partnership v. Commissioner, Docket Nos. 2685-11 and 8393-12, order of 13 July 2016 denying the Commissioner's motion to compel. Copy of the signed order on the eDiscovery Law blog of K&L Gates.","title":"Dynamo Holdings Limited Partnership v. Commissioner, Docket Nos. 2685-11, 8393-12, Order","url":"https://www.ediscoverylaw.com/wp-content/uploads/2016/08/Dynamo-II-Order.pdf","date":"2016-07-13","written_by":"independent","quote":"Petitioners made a reasonable inquiry in responding to the Commissioner's discovery demands when they used predictive coding to produce any documents that the algorithm determined was responsive, and petitioners' response was complete when they produced those documents."}],"replication":{"produced_by":["The Commissioner coded the seed sets that trained the model and chose its recall rate.","The parties agreed on a protocol and filed 16 joint status reports.","A second algorithm screened the production set for privileged material."],"must_hold":["The requesting party takes part in training the model.","The court measures the response by a reasonable inquiry and does not demand perfection."],"echoes":[]},"paired_incident":null,"headline":"Tax Court finds a *predictive coding* response to the IRS complete","metric":"about 406,000 documents into a production set of about 180,000","comparator":null,"later":null,"value":null,"developer_country":null,"deployer_country":"United States","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the United States Tax Court.","source":"https://www.ediscoverylaw.com/wp-content/uploads/2016/08/Dynamo-II-Order.pdf","editor":"AIBD"}],"confidence":"found","shade":"high","claim_status":"Found by a court","claim_basis":"Sources: one independent court document. The cited evidence includes no independent study or report.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0834","catalog_version":"1.0.0","source_fingerprint":"3add5c3a3ca859d7a65e0de027f088d42121227d0569a0f94f47682db8d90c11","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-0834-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Not named in the order","deployer":"Dynamo Holdings Limited Partnership and Beekman Vista, Inc. (petitioners)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"benefit created","ai_role":{"status":"unassessed","statement":"The court judged whether the response met the discovery rules and did not compare the model with a full human review. It assumed for argument that the model's recall fell short and noted a precision of 3%. The model learned from seed sets that the Commissioner coded."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"The taxpayers met the request with a set that the model chose from about 406,000 documents, and the order states no time or cost saved."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"found","claim_status":"Found by a court","basis":"Sources: one independent court document. The cited evidence includes no independent study or report.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"The taxpayers met the request with a set that the model chose from about 406,000 documents, and the order states no time or cost saved."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"The taxpayers met the request with a set that the model chose from about 406,000 documents, and the order states no time or cost saved."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Found by a court. Sources: one independent court document. The cited evidence includes no independent study or report.","evidence_word":"found","claim_status":"Found by a court"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0834-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Found by a court. Sources: one independent court document. The cited evidence includes no independent study or report.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"pilot","answer":"The system was used in a pilot. The review ran once, in one tax case.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0834-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was used in a pilot. The review ran once, in one tax case.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0834"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Repetitive Task Automation"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"After the Tax Court allowed predictive coding in 2014, the parties agreed on a protocol. The Commissioner coded two random sets of 1,000 documents and twelve smaller sets to train and test the model. He chose a recall rate of 95%. The taxpayers ran the model over about 406,000 documents and ran a second algorithm to find privileged material. In January and March 2016 they delivered a production set of about 180,000 documents. The Commissioner kept 5,796 and returned the rest. He then moved to compel documents that his Boolean search had hit and the model had left out. On 13 July 2016 Judge Ronald L. Buch denied the motion. He held that the inquiry was reasonable and that the response was complete. He assumed for argument that the model's recall fell short. He noted that the 95% recall setting brought a precision of 3%. He called the belief that human review is perfect a myth.","benefit_type":"Repetitive Task Automation"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"The taxpayers met the request with a set that the model chose from about 406,000 documents, and the order states no time or cost saved."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"mixed"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"A vendor that the order does not name built the predictive coding software to find responsive documents, with a model trained on the requesting party's coding."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"A vendor that the order does not name built the predictive coding software to find responsive documents, with a model trained on the requesting party's coding."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2016-07-13","date_basis":"source publication","date_precision":"day","revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the United States Tax Court.","source":"https://www.ediscoverylaw.com/wp-content/uploads/2016/08/Dynamo-II-Order.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["The Commissioner coded the seed sets that trained the model and chose its recall rate.","The parties agreed on a protocol and filed 16 joint status reports.","A second algorithm screened the production set for privileged material."],"must_hold":["The requesting party takes part in training the model.","The court measures the response by a reasonable inquiry and does not demand perfection."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the United States Tax Court.","source":"https://www.ediscoverylaw.com/wp-content/uploads/2016/08/Dynamo-II-Order.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Legal services / e-discovery"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"court document","citation":"United States Tax Court (Judge Ronald L. Buch). Dynamo Holdings Limited Partnership v. Commissioner, Docket Nos. 2685-11 and 8393-12, order of 13 July 2016 denying the Commissioner's motion to compel. Copy of the signed order on the eDiscovery Law blog of K&L Gates.","title":"Dynamo Holdings Limited Partnership v. Commissioner, Docket Nos. 2685-11, 8393-12, Order","url":"https://www.ediscoverylaw.com/wp-content/uploads/2016/08/Dynamo-II-Order.pdf","date":"2016-07-13","written_by":"independent","quote":"Petitioners made a reasonable inquiry in responding to the Commissioner's discovery demands when they used predictive coding to produce any documents that the algorithm determined was responsive, and petitioners' response was complete when they produced those documents."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0834-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"court document","citation":"United States Tax Court (Judge Ronald L. Buch). Dynamo Holdings Limited Partnership v. Commissioner, Docket Nos. 2685-11 and 8393-12, order of 13 July 2016 denying the Commissioner's motion to compel. Copy of the signed order on the eDiscovery Law blog of K&L Gates.","title":"Dynamo Holdings Limited Partnership v. Commissioner, Docket Nos. 2685-11, 8393-12, Order","url":"https://www.ediscoverylaw.com/wp-content/uploads/2016/08/Dynamo-II-Order.pdf","date":"2016-07-13","written_by":"independent","quote":"Petitioners made a reasonable inquiry in responding to the Commissioner's discovery demands when they used predictive coding to produce any documents that the algorithm determined was responsive, and petitioners' response was complete when they produced those documents."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In July 2016 the US Tax Court held that a predictive coding response to the IRS was complete. The model sorted about 406,000 documents into a production set of about 180,000, at 3% precision.","account":"After the Tax Court allowed predictive coding in 2014, the parties agreed on a protocol. The Commissioner coded two random sets of 1,000 documents and twelve smaller sets to train and test the model. He chose a recall rate of 95%. The taxpayers ran the model over about 406,000 documents and ran a second algorithm to find privileged material. In January and March 2016 they delivered a production set of about 180,000 documents. The Commissioner kept 5,796 and returned the rest. He then moved to compel documents that his Boolean search had hit and the model had left out. On 13 July 2016 Judge Ronald L. Buch denied the motion. He held that the inquiry was reasonable and that the response was complete. He assumed for argument that the model's recall fell short. He noted that the 95% recall setting brought a precision of 3%. He called the belief that human review is perfect a myth."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Predictive coding algorithm"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"The US Tax Court held that a predictive coding review of about 406,000 documents gave the IRS a complete response, at a precision of 3%","headline":"Tax Court finds a *predictive coding* response to the IRS complete"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["48f1784612c2d8f9266b123daedbcad8edee2b3c195e8fb4ebb83c329a7fec28"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"After the Tax Court allowed predictive coding in 2014, the parties agreed on a protocol. The Commissioner coded two random sets of 1,000 documents and twelve smaller sets to train and test the model. He chose a recall rate of 95%. The taxpayers ran the model over about 406,000 documents and ran a second algorithm to find privileged material. In January and March 2016 they delivered a production set of about 180,000 documents. The Commissioner kept 5,796 and returned the rest. He then moved to compel documents that his Boolean search had hit and the model had left out. On 13 July 2016 Judge Ronald L. Buch denied the motion. He held that the inquiry was reasonable and that the response was complete. He assumed for argument that the model's recall fell short. He noted that the 95% recall setting brought a precision of 3%. He called the belief that human review is perfect a myth.","replication":{"produced_by":["The Commissioner coded the seed sets that trained the model and chose its recall rate.","The parties agreed on a protocol and filed 16 joint status reports.","A second algorithm screened the production set for privileged material."],"must_hold":["The requesting party takes part in training the model.","The court measures the response by a reasonable inquiry and does not demand perfection."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":28,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":26,"review_not_assessed":63,"not_assessed":0,"complete":false,"by_state":{"value_present":28,"not_reported":0,"unresolved":63,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"fb6fd9847439c9db15c567757b21970f77863c7a642e7ba17e4dc85b8e1569eb"}