ExaCT cut data extraction for 75 trials from 21.6 to 17.9 hours

In an evaluation for AHRQ, using ExaCT cut data extraction for 75 randomized trials from 21.6 to 17.9 hours, a modest gain. Reviewers still had to check its output.

  • 1 an organization

    Assisting the first reviewer with ExaCT cut total extraction time across 75 trials from 21.6 to 17.9 hours.

Depth runs from 1, a convenience or time saved, to 5, a life or a livelihood preserved.

Status
Reported benefit.
Evidence

foundstudiedreportedclaimed

An independent evaluation report supports the benefit.

Compared with

Manual data extraction by a single reviewer. Measured.

Without the system
21.6 hours total extraction time across 75 randomized trials
With it
17.9 hours with ExaCT assisting the first reviewer
The system’s part
The evaluation timed manual extraction against the review and amendment of ExaCT's output for the same 75 trials, which isolates the tool's part. The time with ExaCT was measured in a simulated semi-automated workflow, and a human reviewed every extraction.
Limits
Not yet checked.
Costs and harms
Not yet checked.
Later
Not yet checked.

Download data Cite Suggest a correction

Benefit type
Reduced Task Time
System
ExaCT (machine learning and text mining tool for extracting trial data)
Made by
National Research Council of Canada and the University of California, San Francisco
Used by
AHRQ's University of Alberta Evidence-based Practice Center (prospective evaluation on 75 randomized trials)
Where
University of Alberta, Canada
Stage
Tested in a simulation
Date
, when the source was published
Kind
Aggregate result
Broad benefit
Processing Information, Getting Information
Collections
Scrutiny and findings
Last revised
31 October 2026

Sources

  1. Peer-reviewed study, by authors with no tie to the system.

    Gates A, et al. Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool. BMC Medical Research Methodology 2021, 21:169. The authors are the AHRQ report's team at the University of Alberta.

    Developed jointly by the National Research Council of Canada and the University of California, San Francisco, the tool assists reviewers by automatically extracting study characteristics

  2. Evaluation report, by authors with no tie to the system.

    Gates A, et al. Creating Efficiencies in the Extraction of Data From Randomized Trials: A Prospective Evaluation of a Machine Learning and Text Mining Tool. AHRQ Methods for Effective Health Care, Report No. 21-EHC006. Agency for Healthcare Research and Quality, August 2021. NCBI Bookshelf NBK572890, PMID 34403225 (structured abstract).

    Using ExaCT to assist with data extraction resulted in modest gains in efficiency compared with manual extraction. The tool was reliable for identifying the reporting of most data elements.

Details

AI system

Workflow

Reviewers uploaded each trial report to ExaCT and then reviewed and amended its extractions. The tool returned the top sentences for each data element with pertinent fragments marked.

Follow-up

Conditions for replication

A human reviewer checks and amends every extraction, since a median 48% of solutions were fully correct. The data elements are among those that the tool was built to extract.

Entry details

Sector

Evidence synthesis / systematic reviews.

Purpose

The National Research Council of Canada and the University of California, San Francisco built ExaCT to extract study characteristics automatically from publications of randomized trials.

Not yet checked: model, release, provider, role, configuration, inputs and outputs, autonomy, human oversight, beneficiaries, reach, distribution and access, duration, every measurement question, every money question, ownership and control, dependencies, cross-border connections, geopolitical implications, replication attempts, availability, institutional decision, grounds, action and result, success criterion, every evidence question, observed or intended, alternative assessments, related entries, other identifiers, category mappings.

Cite this record

Peter H. Vartanian. (2026). ExaCT cut data extraction for 75 trials from 21.6 to 17.9 hours (AIBD-0749). Artificial Intelligence Benefits Database, edition of 31 October 2026. https://petervartanian.xyz/aibd/records/AIBD-0749.html (archived at https://doi.org/10.7910/DVN/AADSII, version 4.0)

BibTeX Data Field definitions