Abstrackr catches studies that a lone reviewer wrongly excluded

In simulations on three completed reviews, Abstrackr caught 33–90% of records that a single reviewer wrongly excluded and cut screening workload by a median 40%.

  • Not rated an organization

    The gains were measured in retrospective simulations on three completed reviews, and no review team relied on the predictions in live work.

Depth runs from 1, a convenience or time saved, to 5, a life or a livelihood preserved.

Status
Reported and intended benefits.
Evidence

foundstudiedreportedclaimed

An independent evaluation report supports the benefit.

Compared with

A single reviewer's screening without the tool, checked against dual independent screening. Measured.

The system’s part
The simulations compared a single reviewer's screening with and without Abstrackr's predictions against dual independent screening, which isolates the tool's part. The tool learns from the reviewer's labels on a training set of 200 records.
Limits
Not yet checked.
Costs and harms
Not yet checked.
Later
Not yet checked.

Download data Cite Suggest a correction

Benefit type
Improved Task Accuracy
System
Abstrackr (machine learning citation screening)
Made by
Brown University, Center for Evidence Synthesis in Health
Used by
AHRQ's University of Alberta Evidence-based Practice Center (retrospective simulations on three reviews)
Where
University of Alberta, Canada
Stage
Tested in a simulation
Date
, when the source was published
Kind
Aggregate result
Broad benefit
Processing Information, Getting Information
Collections
Scrutiny and findings, Cross-border records
Last revised
31 October 2026

Sources

  1. Evaluation report, by authors with no tie to the system.

    Gates A, et al. Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools. AHRQ Methods for Effective Health Care, Report No. 19(20)-EHC027-EF. Agency for Healthcare Research and Quality, November 2019. NCBI Bookshelf NBK550175, PMID 31790164 (structured abstract).

    Abstrackr identified 33–90% of records erroneously excluded by a single reviewer, while RobotAnalyst performed less well and DistillerSR provided no relative advantage.

  2. Other, by the makers. Context or contrary evidence.

    Brown University School of Public Health, Center for Evidence Synthesis in Health. Technology: turnkey software.

    Are you looking for software to help you streamline and organize your systematic reviews? We've developed these free, open-source programs.

Details

AI system

Workflow

A senior reviewer screened a training set of 200 records in the tool. The tool's relevance predictions for the remaining records were combined with a single reviewer's decisions.

Follow-up

Conditions for replication

The labels in the training set are accurate. A human reviewer still screens every record, since automatic exclusion missed up to 28% of relevant records with Abstrackr.

Entry details

Sector

Evidence synthesis / systematic reviews.

Purpose

Brown University's Center for Evidence Synthesis in Health built Abstrackr to help review teams screen citations and to predict which remaining records are relevant.

Not yet checked: model, release, provider, role, configuration, inputs and outputs, autonomy, human oversight, beneficiaries, reach, distribution and access, duration, every measurement question, every money question, ownership and control, dependencies, cross-border connections, geopolitical implications, replication attempts, availability, institutional decision, grounds, action and result, success criterion, every evidence question, observed or intended, alternative assessments, related entries, other identifiers, category mappings.

Cite this record

Peter H. Vartanian. (2026). Abstrackr catches studies that a lone reviewer wrongly excluded (AIBD-0748). Artificial Intelligence Benefits Database, edition of 31 October 2026. https://petervartanian.xyz/aibd/records/AIBD-0748.html (archived at https://doi.org/10.7910/DVN/AADSII, version 4.0)

BibTeX Data Field definitions