{"id":"AIBD-0749","title":"Assessors for AHRQ find that the ExaCT tool cut the time to extract data from 75 randomized trials from 21.6 to 17.9 hours","summary":"In an evaluation for AHRQ, using ExaCT cut data extraction for 75 randomized trials from 21.6 to 17.9 hours, a modest gain. Reviewers still had to check its output.","date_observed":"2021-08-01","date_basis":"source publication","date_precision":"month","benefit_status":"reported","source_collection":null,"source_entry":null,"account":"The Agency for Healthcare Research and Quality (AHRQ) published an evaluation of ExaCT by its University of Alberta Evidence-based Practice Center in August 2021. ExaCT is an online tool that uses machine learning and text mining to extract data from reports of randomized trials. The team manually extracted and verified 21 data elements from 75 trials published in 2017. They then timed how long reviewers took to review and amend ExaCT's extractions. Using ExaCT to assist the first reviewer cut total extraction time from 21.6 to 17.9 hours. The authors call this a modest gain in efficiency. ExaCT detected whether each data element was reported, with a median accuracy of 91%. A median 48% of its solutions were fully correct, so a human still had to check and amend the output. In a journal article on the same study, the authors report that the National Research Council of Canada and the University of California, San Francisco developed the tool.","category":"aggregate result","unexpected":false,"purpose":"The National Research Council of Canada and the University of California, San Francisco built ExaCT to extract study characteristics automatically from publications of randomized trials.","system":"ExaCT (machine learning and text mining tool for extracting trial data)","model_version":null,"developer":"National Research Council of Canada and the University of California, San Francisco","deployer":"AHRQ's University of Alberta Evidence-based Practice Center (prospective evaluation on 75 randomized trials)","location":"University of Alberta, Canada","domain":"Evidence synthesis / systematic reviews","segment":null,"benefit_type":"Reduced Task Time","work_activities":["4.A.2.a.2","4.A.1.a.1"],"taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"organizational","depth":1,"reason":"Assisting the first reviewer with ExaCT cut total extraction time across 75 trials from 21.6 to 17.9 hours."}],"ai_role":{"status":"assessed","statement":"The evaluation timed manual extraction against the review and amendment of ExaCT's output for the same 75 trials, which isolates the tool's part. The time with ExaCT was measured in a simulated semi-automated workflow, and a human reviewed every extraction."},"evidence":[{"kind":"evaluation report","citation":"Gates A, et al. Creating Efficiencies in the Extraction of Data From Randomized Trials: A Prospective Evaluation of a Machine Learning and Text Mining Tool. AHRQ Methods for Effective Health Care, Report No. 21-EHC006. Agency for Healthcare Research and Quality, August 2021. NCBI Bookshelf NBK572890, PMID 34403225 (structured abstract).","title":"Creating Efficiencies in the Extraction of Data From Randomized Trials: A Prospective Evaluation of a Machine Learning and Text Mining Tool","url":"https://europepmc.org/abstract/MED/34403225","date":null,"written_by":"independent","quote":"Using ExaCT to assist with data extraction resulted in modest gains in efficiency compared with manual extraction. The tool was reliable for identifying the reporting of most data elements."},{"kind":"peer-reviewed study","citation":"Gates A, et al. Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool. BMC Medical Research Methodology 2021, 21:169. The authors are the AHRQ report's team at the University of Alberta.","title":"Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC8369614/","date":"2021-08-16","written_by":"independent","quote":"Developed jointly by the National Research Council of Canada and the University of California, San Francisco, the tool assists reviewers by automatically extracting study characteristics"}],"replication":{"produced_by":["Reviewers uploaded each trial report to ExaCT and then reviewed and amended its extractions.","The tool returned the top sentences for each data element with pertinent fragments marked."],"must_hold":["A human reviewer checks and amends every extraction, since a median 48% of solutions were fully correct.","The data elements are among those that the tool was built to extract."],"echoes":[]},"paired_incident":null,"headline":"*ExaCT* cut data extraction for 75 trials from 21.6 to 17.9 hours","metric":"cut total extraction time from 21.6 to 17.9 hours","comparator":{"against":"Manual data extraction by a single reviewer","before":"21.6 hours total extraction time across 75 randomized trials","after":"17.9 hours with ExaCT assisting the first reviewer","basis":"measured","quote":"Using ExaCT to assist the first reviewer resulted in a modest time savings compared with manual extraction by a single reviewer (17.9 vs. 21.6 hours total extraction time across 75 randomized trials).","url":"https://europepmc.org/abstract/MED/34403225"},"later":null,"value":null,"developer_country":"Canada and United States","deployer_country":"Canada","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the University of Alberta Evidence-based Practice Center.","source":"https://europepmc.org/abstract/MED/34403225","editor":"AIBD"},{"date":"2026-10-31","change":"Rewrote the prose in plain sentences, with its facts, figures and quotations unchanged.","source":null,"editor":"AIBD"}],"confidence":"found","shade":"high","claim_status":"Found by an evaluation office","claim_basis":"Sources: one independent evaluation report and one independent peer-reviewed study.","case_jurisdictions":["Canada"],"profile":{"case_id":"AIBD-0749","catalog_version":"1.0.0","source_fingerprint":"a29f35bff748f172d8ec3142f703d20fd19421402397927a769b586dd1ac8d13","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-0749-source-0001","AIBD-0749-source-0002"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"National Research Council of Canada and the University of California, San Francisco","deployer":"AHRQ's University of Alberta Evidence-based Practice Center (prospective evaluation on 75 randomized trials)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"value_present","value":{"kind":"record_fields","fields":{"ai_role":{"status":"assessed","statement":"The evaluation timed manual extraction against the review and amendment of ExaCT's output for the same 75 trials, which isolates the tool's part. The time with ExaCT was measured in a simulated semi-automated workflow, and a human reviewed every extraction."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"aggregate result","ai_role":{"status":"assessed","statement":"The evaluation timed manual extraction against the review and amendment of ExaCT's output for the same 75 trials, which isolates the tool's part. The time with ExaCT was measured in a simulated semi-automated workflow, and a human reviewed every extraction."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"Assisting the first reviewer with ExaCT cut total extraction time across 75 trials from 21.6 to 17.9 hours."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"found","claim_status":"Found by an evaluation office","basis":"Sources: one independent evaluation report and one independent peer-reviewed study.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"Assisting the first reviewer with ExaCT cut total extraction time across 75 trials from 21.6 to 17.9 hours."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"Assisting the first reviewer with ExaCT cut total extraction time across 75 trials from 21.6 to 17.9 hours."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Found by an evaluation office. Sources: one independent evaluation report and one independent peer-reviewed study.","evidence_word":"found","claim_status":"Found by an evaluation office"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0749-source-0001","AIBD-0749-source-0002"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Found by an evaluation office. Sources: one independent evaluation report and one independent peer-reviewed study.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"simulated","answer":"The benefit was shown in a simulation. The team timed the review of ExaCT's extractions in a simulated semi-automated workflow on 75 trials.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0749-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The benefit was shown in a simulation. The team timed the review of ExaCT's extractions in a simulated semi-automated workflow on 75 trials.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0749"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Reduced Task Time"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"The Agency for Healthcare Research and Quality (AHRQ) published an evaluation of ExaCT by its University of Alberta Evidence-based Practice Center in August 2021. ExaCT is an online tool that uses machine learning and text mining to extract data from reports of randomized trials. The team manually extracted and verified 21 data elements from 75 trials published in 2017. They then timed how long reviewers took to review and amend ExaCT's extractions. Using ExaCT to assist the first reviewer cut total extraction time from 21.6 to 17.9 hours. The authors call this a modest gain in efficiency. ExaCT detected whether each data element was reported, with a median accuracy of 91%. A median 48% of its solutions were fully correct, so a human still had to check and amend the output. In a journal article on the same study, the authors report that the National Research Council of Canada and the University of California, San Francisco developed the tool.","benefit_type":"Reduced Task Time"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"Assisting the first reviewer with ExaCT cut total extraction time across 75 trials from 21.6 to 17.9 hours."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"reported"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"The National Research Council of Canada and the University of California, San Francisco built ExaCT to extract study characteristics automatically from publications of randomized trials."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"University of Alberta, Canada","developer_country":"Canada and United States","deployer_country":"Canada"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"The National Research Council of Canada and the University of California, San Francisco built ExaCT to extract study characteristics automatically from publications of randomized trials."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2021-08-01","date_basis":"source publication","date_precision":"month","revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the University of Alberta Evidence-based Practice Center.","source":"https://europepmc.org/abstract/MED/34403225","editor":"AIBD"},{"date":"2026-10-31","change":"Rewrote the prose in plain sentences, with its facts, figures and quotations unchanged.","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["Reviewers uploaded each trial report to ExaCT and then reviewed and amended its extractions.","The tool returned the top sentences for each data element with pertinent fragments marked."],"must_hold":["A human reviewer checks and amends every extraction, since a median 48% of solutions were fully correct.","The data elements are among those that the tool was built to extract."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the University of Alberta Evidence-based Practice Center.","source":"https://europepmc.org/abstract/MED/34403225","editor":"AIBD"},{"date":"2026-10-31","change":"Rewrote the prose in plain sentences, with its facts, figures and quotations unchanged.","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Evidence synthesis / systematic reviews"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"evaluation report","citation":"Gates A, et al. Creating Efficiencies in the Extraction of Data From Randomized Trials: A Prospective Evaluation of a Machine Learning and Text Mining Tool. AHRQ Methods for Effective Health Care, Report No. 21-EHC006. Agency for Healthcare Research and Quality, August 2021. NCBI Bookshelf NBK572890, PMID 34403225 (structured abstract).","title":"Creating Efficiencies in the Extraction of Data From Randomized Trials: A Prospective Evaluation of a Machine Learning and Text Mining Tool","url":"https://europepmc.org/abstract/MED/34403225","written_by":"independent","quote":"Using ExaCT to assist with data extraction resulted in modest gains in efficiency compared with manual extraction. The tool was reliable for identifying the reporting of most data elements."},{"kind":"peer-reviewed study","citation":"Gates A, et al. Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool. BMC Medical Research Methodology 2021, 21:169. The authors are the AHRQ report's team at the University of Alberta.","title":"Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC8369614/","date":"2021-08-16","written_by":"independent","quote":"Developed jointly by the National Research Council of Canada and the University of California, San Francisco, the tool assists reviewers by automatically extracting study characteristics"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0749-source-0001","AIBD-0749-source-0002"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"evaluation report","citation":"Gates A, et al. Creating Efficiencies in the Extraction of Data From Randomized Trials: A Prospective Evaluation of a Machine Learning and Text Mining Tool. AHRQ Methods for Effective Health Care, Report No. 21-EHC006. Agency for Healthcare Research and Quality, August 2021. NCBI Bookshelf NBK572890, PMID 34403225 (structured abstract).","title":"Creating Efficiencies in the Extraction of Data From Randomized Trials: A Prospective Evaluation of a Machine Learning and Text Mining Tool","url":"https://europepmc.org/abstract/MED/34403225","written_by":"independent","quote":"Using ExaCT to assist with data extraction resulted in modest gains in efficiency compared with manual extraction. The tool was reliable for identifying the reporting of most data elements."},{"kind":"peer-reviewed study","citation":"Gates A, et al. Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool. BMC Medical Research Methodology 2021, 21:169. The authors are the AHRQ report's team at the University of Alberta.","title":"Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool","url":"https://pmc.ncbi.nlm.nih.gov/articles/PMC8369614/","date":"2021-08-16","written_by":"independent","quote":"Developed jointly by the National Research Council of Canada and the University of California, San Francisco, the tool assists reviewers by automatically extracting study characteristics"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In an evaluation for AHRQ, using ExaCT cut data extraction for 75 randomized trials from 21.6 to 17.9 hours, a modest gain. Reviewers still had to check its output.","account":"The Agency for Healthcare Research and Quality (AHRQ) published an evaluation of ExaCT by its University of Alberta Evidence-based Practice Center in August 2021. ExaCT is an online tool that uses machine learning and text mining to extract data from reports of randomized trials. The team manually extracted and verified 21 data elements from 75 trials published in 2017. They then timed how long reviewers took to review and amend ExaCT's extractions. Using ExaCT to assist the first reviewer cut total extraction time from 21.6 to 17.9 hours. The authors call this a modest gain in efficiency. ExaCT detected whether each data element was reported, with a median accuracy of 91%. A median 48% of its solutions were fully correct, so a human still had to check and amend the output. In a journal article on the same study, the authors report that the National Research Council of Canada and the University of California, San Francisco developed the tool."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"ExaCT (machine learning and text mining tool for extracting trial data)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"Assessors for AHRQ find that the ExaCT tool cut the time to extract data from 75 randomized trials from 21.6 to 17.9 hours","headline":"*ExaCT* cut data extraction for 75 trials from 21.6 to 17.9 hours"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["d0df0b49ef60d6eadd8eb2e7f929aa85c7dfd06f92e340929bb2dfe3373ac955"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"The Agency for Healthcare Research and Quality (AHRQ) published an evaluation of ExaCT by its University of Alberta Evidence-based Practice Center in August 2021. ExaCT is an online tool that uses machine learning and text mining to extract data from reports of randomized trials. The team manually extracted and verified 21 data elements from 75 trials published in 2017. They then timed how long reviewers took to review and amend ExaCT's extractions. Using ExaCT to assist the first reviewer cut total extraction time from 21.6 to 17.9 hours. The authors call this a modest gain in efficiency. ExaCT detected whether each data element was reported, with a median accuracy of 91%. A median 48% of its solutions were fully correct, so a human still had to check and amend the output. In a journal article on the same study, the authors report that the National Research Council of Canada and the University of California, San Francisco developed the tool.","replication":{"produced_by":["Reviewers uploaded each trial report to ExaCT and then reviewed and amended its extractions.","The tool returned the top sentences for each data element with pertinent fragments marked."],"must_hold":["A human reviewer checks and amends every extraction, since a median 48% of solutions were fully correct.","The data elements are among those that the tool was built to extract."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":29,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":27,"review_not_assessed":62,"not_assessed":0,"complete":false,"by_state":{"value_present":29,"not_reported":0,"unresolved":62,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"ec9ab9b801895e9df17a62d108ab961095f0beda484296af58c74ffc83399192"}