{"id":"AIBD-0668","title":"Vincent AI and OpenAI o1-preview raise the quality and productivity of law students' legal work in a randomized trial (Schwarcz et al.)","summary":"In a randomized trial with 137 law students, Vincent AI and o1-preview raised the quality of legal work and raised productivity on five of six tasks by 50% to 130%.","date_observed":"2024-10-31","date_basis":"source information","date_precision":"day","benefit_status":"reported","source_collection":null,"source_entry":null,"account":"Daniel Schwarcz and David Cleveland of the University of Minnesota, Sam Manning of GovAI, J.J. Prescott and Patrick Barry of the University of Michigan, and Beverly Rich of the University of Southern California ran a randomized trial with upper-level law students at the Michigan and Minnesota law schools from 1 to 31 October 2024. Of 153 students who enrolled, 137 completed at least one task and 125 completed all six. Each student did six legal tasks, some with Vincent AI, a retrieval-augmented legal research tool from vLex, some with OpenAI's o1-preview reasoning model and some with no AI; random assignment to one of three groups decided which tasks went with which condition. Three of the authors graded the work blind. The authors report that both tools raised the quality of the legal work and that they raised productivity by a statistically significant margin on five of the six tasks, with gains of 50% to 130%. Both tools cut time use by about 20% to 28% on five tasks. The tools did best on complex tasks such as drafting persuasive letters and analyzing complaints. Hallucinated citations were rare: the authors counted 3 in work done with Vincent AI, 4 in work done without AI and 11 in work done with o1-preview. Law firms and the two law schools funded the study, and the authors write that neither OpenAI nor vLex gave financial support, although both gave participants free access. The Journal of Law and Empirical Analysis published the study on 9 April 2026.","category":"general benefit","unexpected":false,"purpose":"Vincent AI was built for legal research and drafting, and o1-preview is a general reasoning model. Better and faster legal work was the purpose that the trial tested.","system":"Vincent AI (vLex, retrieval-augmented generation) and OpenAI o1-preview","model_version":"o1-preview; Vincent AI as of 2024","developer":"vLex (Vincent AI); OpenAI (o1-preview)","deployer":"Law school researchers at the University of Minnesota and the University of Michigan","location":"University of Michigan Law School and University of Minnesota Law School, United States (students worked remotely)","domain":"Legal services","segment":null,"benefit_type":"Improved Task Accuracy","taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"personal","depth":2,"reason":"Students using either tool produced legal work of higher graded quality and were 50% to 130% more productive on five of six tasks."}],"ai_role":{"status":"assessed","statement":"Students were randomly assigned to one of the two tools or to no AI, so the differences isolate the effect of each tool."},"evidence":[{"kind":"peer-reviewed study","citation":"Schwarcz D, Manning S, Prescott JJ, Barry P, Cleveland DR, Rich B. AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice. Journal of Law and Empirical Analysis. 2026;3(1):220-250. doi:10.1177/2755323X261427048 (University of Minnesota repository copy, archived).","title":"AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice","url":"https://web.archive.org/web/2026id_/https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=2220&context=faculty_articles","date":"2026-04-09","written_by":"independent","quote":"Our findings also show that these AI tools significantly boost productivity in five out of six tested legal tasks, with statistically significant gains of anywhere from 50% to 130%."},{"kind":"preprint","citation":"Schwarcz D, Manning S, Barry P, Cleveland DR, Prescott JJ, Rich B. AI-Powered Lawyering. Minnesota Legal Studies Research Paper 25-16, 2 March 2025 (SSRN 5162111, archived abstract page).","title":"AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice","url":"https://web.archive.org/web/2025/https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5162111","date":"2025-03-02","written_by":"independent","quote":"We find that both AI tools significantly enhance legal work quality, a marked contrast with previous research examining older large language models like GPT-4."}],"replication":{"produced_by":["Students used either a retrieval-augmented legal tool or a reasoning model on realistic legal tasks.","Students received training in using both tools before the tasks.","Graders scored the work blind to the condition."],"must_hold":["The tool must ground its analysis in legal sources or reason through the problem before writing.","Users must check citations for hallucinations."],"echoes":[{"date":"2023-01-01","setting":"University of Minnesota law students with GPT-4 (speed gains, little quality gain)","model_version":null,"outcome":"recurred","source":"https://doi.org/10.24926/265535.4225"},{"date":"2025-10-27","setting":"133 patent lawyers at eleven U.S. firms with an AI patent drafting assistant","model_version":null,"outcome":"recurred","source":"https://www.nber.org/papers/w35720"}]},"paired_incident":{"database":"AI Incident Database","identifier":"1572","title":"Westlaw CoCounsel Reportedly Generated False Legal Quotations in United States v. Farris","url":"https://incidentdatabase.ai/cite/1572","severity":null,"group":"personal","basis":"Both cases concern a legal research AI tool built on retrieval-augmented generation that a person uses to prepare legal work, where the person is held responsible for false quotations or citations in the result."},"headline":"Two legal AI tools improve students' legal work in a trial","metric":"raised productivity on five of six tasks by 50% to 130%","comparator":{"against":"Students completing the same tasks without AI","before":"Productivity and graded quality without AI","after":"Higher quality; productivity gains of 50% to 130% on five of six tasks","basis":"measured","quote":"Our findings also show that these AI tools significantly boost productivity in five out of six tested legal tasks","url":"https://web.archive.org/web/2026id_/https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=2220&context=faculty_articles"},"later":null,"value":null,"developer_country":"United States (OpenAI); not verified for vLex","deployer_country":"United States","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://web.archive.org/web/2026id_/https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=2220&context=faculty_articles","editor":"AIBD"},{"date":"2026-10-29","change":"Linked a comparable incident in the AI Incident Database (incident 1572).","source":"https://incidentdatabase.ai/cite/1572","editor":"AIBD"}],"confidence":"studied","shade":"high","claim_status":"Independently studied","claim_basis":"Sources: one independent peer-reviewed study and one independent preprint. The cited evidence includes no finding by a regulator or court.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0668","catalog_version":"1.0.0","source_fingerprint":"b11d1becfe203841a6cede2f6807b64f36392e8090060d8f3cef741e07bca158","collection_refs":[{"collection_id":"aibd","version":"2026-10-29"}],"source_ids":["AIBD-0668-source-0001","AIBD-0668-source-0002"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0668-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":[],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"partial","record_ids":["AIBD-0668-evaluation-1"]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"vLex (Vincent AI); OpenAI (o1-preview)","deployer":"Law school researchers at the University of Minnesota and the University of Michigan"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"value_present","value":{"kind":"record_fields","fields":{"ai_role":{"status":"assessed","statement":"Students were randomly assigned to one of the two tools or to no AI, so the differences isolate the effect of each tool."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"general benefit","ai_role":{"status":"assessed","statement":"Students were randomly assigned to one of the two tools or to no AI, so the differences isolate the effect of each tool."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Students using either tool produced legal work of higher graded quality and were 50% to 130% more productive on five of six tasks."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"studied","claim_status":"Independently studied","basis":"Sources: one independent peer-reviewed study and one independent preprint. The cited evidence includes no finding by a regulator or court.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Students using either tool produced legal work of higher graded quality and were 50% to 130% more productive on five of six tasks."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Students using either tool produced legal work of higher graded quality and were 50% to 130% more productive on five of six tasks."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"value_present","value":{"kind":"structured","data":{"classification":"evaluation or study","answer":"The case describes an evaluation or a study. A randomized trial.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The case describes an evaluation or a study. A randomized trial.","inferred":false}},"case_links[]":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Both cases concern a legal research AI tool built on retrieval-augmented generation that a person uses to prepare legal work, where the person is held responsible for false quotations or citations in the result.","links":[{"database":"AI Incident Database","identifier":"1572","title":"Westlaw CoCounsel Reportedly Generated False Legal Quotations in United States v. Farris","url":"https://incidentdatabase.ai/cite/1572","group":"personal","basis":"Both cases concern a legal research AI tool built on retrieval-augmented generation that a person uses to prepare legal work, where the person is held responsible for false quotations or citations in the result.","relation":"comparable_harm"}]}},"provenance":{"kind":"curatorial","basis":"Compared with the incidents in the AI Incident Database snapshot of 26 October 2026; a second reading confirmed that the incident concerns the same kind of system and stakeholder.","record_fields":["paired_incident"],"source_refs":["https://incidentdatabase.ai/cite/1572"]},"review_state":"reviewed","coverage":"complete","display":{"answer":"Both cases concern a legal research AI tool built on retrieval-augmented generation that a person uses to prepare legal work, where the person is held responsible for false quotations or citations in the result.","inferred":false}},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Independently studied. Sources: one independent peer-reviewed study and one independent preprint. The cited evidence includes no finding by a regulator or court.","evidence_word":"studied","claim_status":"Independently studied"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001","AIBD-0668-source-0002"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Independently studied. Sources: one independent peer-reviewed study and one independent preprint. The cited evidence includes no finding by a regulator or court.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"laboratory test","answer":"The system was tested in a laboratory. Students completed designed tasks in a study.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was tested in a laboratory. Students completed designed tasks in a study.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"value_present","value":{"kind":"text","text":"Productivity up 50% to 130% on five tasks."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Productivity up 50% to 130% on five tasks.","inferred":false}},"evaluations[].comparator":{"state":"value_present","value":{"kind":"text","text":"No AI."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"No AI.","inferred":false}},"evaluations[].design":{"state":"value_present","value":{"kind":"text","text":"Randomized."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Randomized.","inferred":false}},"evaluations[].effect":{"state":"value_present","value":{"kind":"text","text":"Significant quality gains for both tools; productivity gains of 50% to 130% on five of six tasks."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Significant quality gains for both tools; productivity gains of 50% to 130% on five of six tasks.","inferred":false}},"evaluations[].exposure":{"state":"value_present","value":{"kind":"text","text":"Vincent AI or o1-preview during the tasks."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Vincent AI or o1-preview during the tasks.","inferred":false}},"evaluations[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0668-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Randomized.","inferred":false}},"evaluations[].measurement_method":{"state":"value_present","value":{"kind":"text","text":"Graded quality and productivity on six legal tasks."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Graded quality and productivity on six legal tasks.","inferred":false}},"evaluations[].metric":{"state":"value_present","value":{"kind":"text","text":"Quality grades and productivity."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Quality grades and productivity.","inferred":false}},"evaluations[].reproduction_materials":{"state":"value_present","value":{"kind":"text","text":"Task texts in Appendix B of the article."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Task texts in Appendix B of the article.","inferred":false}},"evaluations[].setting_sample":{"state":"value_present","value":{"kind":"text","text":"153 enrolled upper-level law students at Michigan and Minnesota; 137 completed at least one task, 125 all six."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"153 enrolled upper-level law students at Michigan and Minnesota; 137 completed at least one task, 125 all six.","inferred":false}},"evaluations[].uncertainty_limitations":{"state":"value_present","value":{"kind":"text","text":"Students in place of practicing lawyers; designed tasks; graders were co-authors; o1-preview work had more hallucinated citations."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0668-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Students in place of practicing lawyers; designed tasks; graders were co-authors; o1-preview work had more hallucinated citations.","inferred":false}},"external_references[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"paired_incident":{"database":"AI Incident Database","identifier":"1572","title":"Westlaw CoCounsel Reportedly Generated False Legal Quotations in United States v. Farris","url":"https://incidentdatabase.ai/cite/1572","group":"personal","basis":"Both cases concern a legal research AI tool built on retrieval-augmented generation that a person uses to prepare legal work, where the person is held responsible for false quotations or citations in the result."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["paired_incident"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0668"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Daniel Schwarcz and David Cleveland of the University of Minnesota, Sam Manning of GovAI, J.J. Prescott and Patrick Barry of the University of Michigan, and Beverly Rich of the University of Southern California ran a randomized trial with upper-level law students at the Michigan and Minnesota law schools from 1 to 31 October 2024. Of 153 students who enrolled, 137 completed at least one task and 125 completed all six. Each student did six legal tasks, some with Vincent AI, a retrieval-augmented legal research tool from vLex, some with OpenAI's o1-preview reasoning model and some with no AI; random assignment to one of three groups decided which tasks went with which condition. Three of the authors graded the work blind. The authors report that both tools raised the quality of the legal work and that they raised productivity by a statistically significant margin on five of the six tasks, with gains of 50% to 130%. Both tools cut time use by about 20% to 28% on five tasks. The tools did best on complex tasks such as drafting persuasive letters and analyzing complaints. Hallucinated citations were rare: the authors counted 3 in work done with Vincent AI, 4 in work done without AI and 11 in work done with o1-preview. Law firms and the two law schools funded the study, and the authors write that neither OpenAI nor vLex gave financial support, although both gave participants free access. The Journal of Law and Empirical Analysis published the study on 9 April 2026.","benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Students using either tool produced legal work of higher graded quality and were 50% to 130% more productive on five of six tasks."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"reported"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"Vincent AI was built for legal research and drafting, and o1-preview is a general reasoning model. Better and faster legal work was the purpose that the trial tested."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"University of Michigan Law School and University of Minnesota Law School, United States (students worked remotely)","developer_country":"United States (OpenAI); not verified for vLex","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"Vincent AI was built for legal research and drafting, and o1-preview is a general reasoning model. Better and faster legal work was the purpose that the trial tested."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2024-10-31","date_basis":"source information","date_precision":"day","revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://web.archive.org/web/2026id_/https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=2220&context=faculty_articles","editor":"AIBD"},{"date":"2026-10-29","change":"Linked a comparable incident in the AI Incident Database (incident 1572).","source":"https://incidentdatabase.ai/cite/1572","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["Students used either a retrieval-augmented legal tool or a reasoning model on realistic legal tasks.","Students received training in using both tools before the tasks.","Graders scored the work blind to the condition."],"must_hold":["The tool must ground its analysis in legal sources or reason through the problem before writing.","Users must check citations for hallucinations."],"echoes":[{"date":"2023-01-01","setting":"University of Minnesota law students with GPT-4 (speed gains, little quality gain)","outcome":"recurred","source":"https://doi.org/10.24926/265535.4225"},{"date":"2025-10-27","setting":"133 patent lawyers at eleven U.S. firms with an AI patent drafting assistant","outcome":"recurred","source":"https://www.nber.org/papers/w35720"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["Students used either a retrieval-augmented legal tool or a reasoning model on realistic legal tasks.","Students received training in using both tools before the tasks.","Graders scored the work blind to the condition."],"must_hold":["The tool must ground its analysis in legal sources or reason through the problem before writing.","Users must check citations for hallucinations."],"echoes":[{"date":"2023-01-01","setting":"University of Minnesota law students with GPT-4 (speed gains, little quality gain)","outcome":"recurred","source":"https://doi.org/10.24926/265535.4225"},{"date":"2025-10-27","setting":"133 patent lawyers at eleven U.S. firms with an AI patent drafting assistant","outcome":"recurred","source":"https://www.nber.org/papers/w35720"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://web.archive.org/web/2026id_/https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=2220&context=faculty_articles","editor":"AIBD"},{"date":"2026-10-29","change":"Linked a comparable incident in the AI Incident Database (incident 1572).","source":"https://incidentdatabase.ai/cite/1572","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Legal services"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"peer-reviewed study","citation":"Schwarcz D, Manning S, Prescott JJ, Barry P, Cleveland DR, Rich B. AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice. Journal of Law and Empirical Analysis. 2026;3(1):220-250. doi:10.1177/2755323X261427048 (University of Minnesota repository copy, archived).","title":"AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice","url":"https://web.archive.org/web/2026id_/https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=2220&context=faculty_articles","date":"2026-04-09","written_by":"independent","quote":"Our findings also show that these AI tools significantly boost productivity in five out of six tested legal tasks, with statistically significant gains of anywhere from 50% to 130%."},{"kind":"preprint","citation":"Schwarcz D, Manning S, Barry P, Cleveland DR, Prescott JJ, Rich B. AI-Powered Lawyering. Minnesota Legal Studies Research Paper 25-16, 2 March 2025 (SSRN 5162111, archived abstract page).","title":"AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice","url":"https://web.archive.org/web/2025/https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5162111","date":"2025-03-02","written_by":"independent","quote":"We find that both AI tools significantly enhance legal work quality, a marked contrast with previous research examining older large language models like GPT-4."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0668-source-0001","AIBD-0668-source-0002"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"peer-reviewed study","citation":"Schwarcz D, Manning S, Prescott JJ, Barry P, Cleveland DR, Rich B. AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice. Journal of Law and Empirical Analysis. 2026;3(1):220-250. doi:10.1177/2755323X261427048 (University of Minnesota repository copy, archived).","title":"AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice","url":"https://web.archive.org/web/2026id_/https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article=2220&context=faculty_articles","date":"2026-04-09","written_by":"independent","quote":"Our findings also show that these AI tools significantly boost productivity in five out of six tested legal tasks, with statistically significant gains of anywhere from 50% to 130%."},{"kind":"preprint","citation":"Schwarcz D, Manning S, Barry P, Cleveland DR, Prescott JJ, Rich B. AI-Powered Lawyering. Minnesota Legal Studies Research Paper 25-16, 2 March 2025 (SSRN 5162111, archived abstract page).","title":"AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice","url":"https://web.archive.org/web/2025/https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5162111","date":"2025-03-02","written_by":"independent","quote":"We find that both AI tools significantly enhance legal work quality, a marked contrast with previous research examining older large language models like GPT-4."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In a randomized trial with 137 law students, Vincent AI and o1-preview raised the quality of legal work and raised productivity on five of six tasks by 50% to 130%.","account":"Daniel Schwarcz and David Cleveland of the University of Minnesota, Sam Manning of GovAI, J.J. Prescott and Patrick Barry of the University of Michigan, and Beverly Rich of the University of Southern California ran a randomized trial with upper-level law students at the Michigan and Minnesota law schools from 1 to 31 October 2024. Of 153 students who enrolled, 137 completed at least one task and 125 completed all six. Each student did six legal tasks, some with Vincent AI, a retrieval-augmented legal research tool from vLex, some with OpenAI's o1-preview reasoning model and some with no AI; random assignment to one of three groups decided which tasks went with which condition. Three of the authors graded the work blind. The authors report that both tools raised the quality of the legal work and that they raised productivity by a statistically significant margin on five of the six tasks, with gains of 50% to 130%. Both tools cut time use by about 20% to 28% on five tasks. The tools did best on complex tasks such as drafting persuasive letters and analyzing complaints. Hallucinated citations were rare: the authors counted 3 in work done with Vincent AI, 4 in work done without AI and 11 in work done with o1-preview. Law firms and the two law schools funded the study, and the authors write that neither OpenAI nor vLex gave financial support, although both gave participants free access. The Journal of Law and Empirical Analysis published the study on 9 April 2026."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"value_present","value":{"kind":"record_fields","fields":{"model_version":"o1-preview; Vincent AI as of 2024"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["model_version"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Vincent AI (vLex, retrieval-augmented generation) and OpenAI o1-preview"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"Vincent AI and OpenAI o1-preview raise the quality and productivity of law students' legal work in a randomized trial (Schwarcz et al.)","headline":"Two legal AI tools improve students' legal work in a trial"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["a54423ecfe89a5e89bf1c0e55640ebec423e939b20605fa496c6a0482de26b04"],"role":"SHA-256 of the complete public case and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public case, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Daniel Schwarcz and David Cleveland of the University of Minnesota, Sam Manning of GovAI, J.J. Prescott and Patrick Barry of the University of Michigan, and Beverly Rich of the University of Southern California ran a randomized trial with upper-level law students at the Michigan and Minnesota law schools from 1 to 31 October 2024. Of 153 students who enrolled, 137 completed at least one task and 125 completed all six. Each student did six legal tasks, some with Vincent AI, a retrieval-augmented legal research tool from vLex, some with OpenAI's o1-preview reasoning model and some with no AI; random assignment to one of three groups decided which tasks went with which condition. Three of the authors graded the work blind. The authors report that both tools raised the quality of the legal work and that they raised productivity by a statistically significant margin on five of the six tasks, with gains of 50% to 130%. Both tools cut time use by about 20% to 28% on five tasks. The tools did best on complex tasks such as drafting persuasive letters and analyzing complaints. Hallucinated citations were rare: the authors counted 3 in work done with Vincent AI, 4 in work done without AI and 11 in work done with o1-preview. Law firms and the two law schools funded the study, and the authors write that neither OpenAI nor vLex gave financial support, although both gave participants free access. The Journal of Law and Empirical Analysis published the study on 9 April 2026.","replication":{"produced_by":["Students used either a retrieval-augmented legal tool or a reasoning model on realistic legal tasks.","Students received training in using both tools before the tasks.","Graders scored the work blind to the condition."],"must_hold":["The tool must ground its analysis in legal sources or reason through the problem before writing.","Users must check citations for hallucinations."],"echoes":[{"date":"2023-01-01","setting":"University of Minnesota law students with GPT-4 (speed gains, little quality gain)","outcome":"recurred","source":"https://doi.org/10.24926/265535.4225"},{"date":"2025-10-27","setting":"133 patent lawyers at eleven U.S. firms with an AI patent drafting assistant","outcome":"recurred","source":"https://www.nber.org/papers/w35720"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":45,"reviewed":15,"reviewed_complete":15,"reviewed_partial":0,"record_mapped":30,"review_not_assessed":46,"not_assessed":0,"complete":false,"by_state":{"value_present":45,"not_reported":0,"unresolved":46,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"b82937a21946257fe403393cdb259643bb7abb348f49a704ff8f0e83e991ef98"}