{"id":"AIBD-0748","title":"Assessors for AHRQ find that Abstrackr's predictions caught 33–90% of records that a single reviewer wrongly excluded in screening simulations","summary":"In simulations on three completed reviews, Abstrackr caught 33–90% of records that a single reviewer wrongly excluded and cut screening workload by a median 40%.","date_observed":"2019-11-01","date_basis":"source publication","date_precision":"month","benefit_status":"mixed","source_collection":null,"source_entry":null,"account":"The Agency for Healthcare Research and Quality (AHRQ) published a methods report by its University of Alberta Evidence-based Practice Center in November 2019. The authors tested three tools that use machine learning to screen records for systematic reviews. They ran retrospective simulations on three reviews. In each tool they screened a training set of 200 records, and the tool predicted the relevance of the rest. In the semi-automated simulation, the predictions complemented the work of a single reviewer. There, Abstrackr identified 33–90% of records that a single reviewer had wrongly excluded. A median 1% of relevant records were missed, and the workload fell by a median 40%. RobotAnalyst performed less well, and DistillerSR offered no relative advantage. Research staff rated Abstrackr usable on the System Usability Scale. When the tools excluded records automatically, Abstrackr missed a median 5% of relevant records and up to 28%.","category":"aggregate result","unexpected":false,"purpose":"Brown University's Center for Evidence Synthesis in Health built Abstrackr to help review teams screen citations and to predict which remaining records are relevant.","system":"Abstrackr (machine learning citation screening)","model_version":null,"developer":"Brown University, Center for Evidence Synthesis in Health","deployer":"AHRQ's University of Alberta Evidence-based Practice Center (retrospective simulations on three reviews)","location":"University of Alberta, Canada","domain":"Evidence synthesis / systematic reviews","segment":null,"benefit_type":"Improved Task Accuracy","work_activities":["4.A.2.a.2","4.A.1.a.1"],"taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"organizational","depth":null,"reason":"The gains were measured in retrospective simulations on three completed reviews, and no review team relied on the predictions in live work."}],"ai_role":{"status":"assessed","statement":"The simulations compared a single reviewer's screening with and without Abstrackr's predictions against dual independent screening, which isolates the tool's part. The tool learns from the reviewer's labels on a training set of 200 records."},"evidence":[{"kind":"evaluation report","citation":"Gates A, et al. Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools. AHRQ Methods for Effective Health Care, Report No. 19(20)-EHC027-EF. Agency for Healthcare Research and Quality, November 2019. NCBI Bookshelf NBK550175, PMID 31790164 (structured abstract).","title":"Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools","url":"https://europepmc.org/abstract/MED/31790164","date":null,"written_by":"independent","quote":"Abstrackr identified 33–90% of records erroneously excluded by a single reviewer, while RobotAnalyst performed less well and DistillerSR provided no relative advantage."},{"kind":"other","citation":"Brown University School of Public Health, Center for Evidence Synthesis in Health. Technology: turnkey software.","title":"Technology | CESH | School of Public Health | Brown University","url":"https://www.brown.edu/public-health/cesh/resources/software","date":null,"written_by":"makers","quote":"Are you looking for software to help you streamline and organize your systematic reviews? We've developed these free, open-source programs.","supports_benefit":false}],"replication":{"produced_by":["A senior reviewer screened a training set of 200 records in the tool.","The tool's relevance predictions for the remaining records were combined with a single reviewer's decisions."],"must_hold":["The labels in the training set are accurate.","A human reviewer still screens every record, since automatic exclusion missed up to 28% of relevant records with Abstrackr."],"echoes":[]},"paired_incident":null,"headline":"*Abstrackr* catches studies that a lone reviewer wrongly excluded","metric":"33–90% of records that a single reviewer wrongly excluded","comparator":{"against":"A single reviewer's screening without the tool, checked against dual independent screening","before":null,"after":"33–90% of records erroneously excluded by a single reviewer identified","basis":"measured","quote":"Abstrackr identified 33–90% of records erroneously excluded by a single reviewer, while RobotAnalyst performed less well and DistillerSR provided no relative advantage.","url":"https://europepmc.org/abstract/MED/31790164"},"later":null,"value":null,"developer_country":"United States","deployer_country":"Canada","unrated_because":"The tool was tested in retrospective simulations, so no review is reported to have gained from it.","brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the University of Alberta Evidence-based Practice Center.","source":"https://europepmc.org/abstract/MED/31790164","editor":"AIBD"}],"confidence":"found","shade":"high","claim_status":"Found by an evaluation office","claim_basis":"Sources: one independent evaluation report and the makers' other source. For the benefit itself, the cited sources include no independent study or report.","case_jurisdictions":["Canada"],"profile":{"case_id":"AIBD-0748","catalog_version":"1.0.0","source_fingerprint":"2a8ef41bd7947ebddb581c4236de33d9840d670b5cb42bb50d53add5460097ae","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"},{"collection_id":"viator","version":"2026-10-25"}],"source_ids":["AIBD-0748-source-0001","AIBD-0748-source-0002"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Brown University, Center for Evidence Synthesis in Health","deployer":"AHRQ's University of Alberta Evidence-based Practice Center (retrospective simulations on three reviews)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"value_present","value":{"kind":"record_fields","fields":{"ai_role":{"status":"assessed","statement":"The simulations compared a single reviewer's screening with and without Abstrackr's predictions against dual independent screening, which isolates the tool's part. The tool learns from the reviewer's labels on a training set of 200 records."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"aggregate result","ai_role":{"status":"assessed","statement":"The simulations compared a single reviewer's screening with and without Abstrackr's predictions against dual independent screening, which isolates the tool's part. The tool learns from the reviewer's labels on a training set of 200 records."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"found","claim_status":"Found by an evaluation office","basis":"Sources: one independent evaluation report and the makers' other source. For the benefit itself, the cited sources include no independent study or report.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The gains were measured in retrospective simulations on three completed reviews, and no review team relied on the predictions in live work."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The gains were measured in retrospective simulations on three completed reviews, and no review team relied on the predictions in live work."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Found by an evaluation office. Sources: one independent evaluation report and the makers' other source. For the benefit itself, the cited sources include no independent study or report.","evidence_word":"found","claim_status":"Found by an evaluation office"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0748-source-0001","AIBD-0748-source-0002"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Found by an evaluation office. Sources: one independent evaluation report and the makers' other source. For the benefit itself, the cited sources include no independent study or report.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"simulated","answer":"The benefit was shown in a simulation. The authors ran retrospective screening simulations on three systematic reviews.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0748-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The benefit was shown in a simulation. The authors ran retrospective screening simulations on three systematic reviews.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0748"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"The Agency for Healthcare Research and Quality (AHRQ) published a methods report by its University of Alberta Evidence-based Practice Center in November 2019. The authors tested three tools that use machine learning to screen records for systematic reviews. They ran retrospective simulations on three reviews. In each tool they screened a training set of 200 records, and the tool predicted the relevance of the rest. In the semi-automated simulation, the predictions complemented the work of a single reviewer. There, Abstrackr identified 33–90% of records that a single reviewer had wrongly excluded. A median 1% of relevant records were missed, and the workload fell by a median 40%. RobotAnalyst performed less well, and DistillerSR offered no relative advantage. Research staff rated Abstrackr usable on the System Usability Scale. When the tools excluded records automatically, Abstrackr missed a median 5% of relevant records and up to 28%.","benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The gains were measured in retrospective simulations on three completed reviews, and no review team relied on the predictions in live work."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"mixed"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"Brown University's Center for Evidence Synthesis in Health built Abstrackr to help review teams screen citations and to predict which remaining records are relevant."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"University of Alberta, Canada","developer_country":"United States","deployer_country":"Canada"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"Brown University's Center for Evidence Synthesis in Health built Abstrackr to help review teams screen citations and to predict which remaining records are relevant."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2019-11-01","date_basis":"source publication","date_precision":"month","revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the University of Alberta Evidence-based Practice Center.","source":"https://europepmc.org/abstract/MED/31790164","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["A senior reviewer screened a training set of 200 records in the tool.","The tool's relevance predictions for the remaining records were combined with a single reviewer's decisions."],"must_hold":["The labels in the training set are accurate.","A human reviewer still screens every record, since automatic exclusion missed up to 28% of relevant records with Abstrackr."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the University of Alberta Evidence-based Practice Center.","source":"https://europepmc.org/abstract/MED/31790164","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Evidence synthesis / systematic reviews"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"evaluation report","citation":"Gates A, et al. Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools. AHRQ Methods for Effective Health Care, Report No. 19(20)-EHC027-EF. Agency for Healthcare Research and Quality, November 2019. NCBI Bookshelf NBK550175, PMID 31790164 (structured abstract).","title":"Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools","url":"https://europepmc.org/abstract/MED/31790164","written_by":"independent","quote":"Abstrackr identified 33–90% of records erroneously excluded by a single reviewer, while RobotAnalyst performed less well and DistillerSR provided no relative advantage."},{"kind":"other","citation":"Brown University School of Public Health, Center for Evidence Synthesis in Health. Technology: turnkey software.","title":"Technology | CESH | School of Public Health | Brown University","url":"https://www.brown.edu/public-health/cesh/resources/software","written_by":"makers","quote":"Are you looking for software to help you streamline and organize your systematic reviews? We've developed these free, open-source programs.","supports_benefit":false}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0748-source-0001","AIBD-0748-source-0002"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"evaluation report","citation":"Gates A, et al. Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools. AHRQ Methods for Effective Health Care, Report No. 19(20)-EHC027-EF. Agency for Healthcare Research and Quality, November 2019. NCBI Bookshelf NBK550175, PMID 31790164 (structured abstract).","title":"Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools","url":"https://europepmc.org/abstract/MED/31790164","written_by":"independent","quote":"Abstrackr identified 33–90% of records erroneously excluded by a single reviewer, while RobotAnalyst performed less well and DistillerSR provided no relative advantage."},{"kind":"other","citation":"Brown University School of Public Health, Center for Evidence Synthesis in Health. Technology: turnkey software.","title":"Technology | CESH | School of Public Health | Brown University","url":"https://www.brown.edu/public-health/cesh/resources/software","written_by":"makers","quote":"Are you looking for software to help you streamline and organize your systematic reviews? We've developed these free, open-source programs.","supports_benefit":false}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In simulations on three completed reviews, Abstrackr caught 33–90% of records that a single reviewer wrongly excluded and cut screening workload by a median 40%.","account":"The Agency for Healthcare Research and Quality (AHRQ) published a methods report by its University of Alberta Evidence-based Practice Center in November 2019. The authors tested three tools that use machine learning to screen records for systematic reviews. They ran retrospective simulations on three reviews. In each tool they screened a training set of 200 records, and the tool predicted the relevance of the rest. In the semi-automated simulation, the predictions complemented the work of a single reviewer. There, Abstrackr identified 33–90% of records that a single reviewer had wrongly excluded. A median 1% of relevant records were missed, and the workload fell by a median 40%. RobotAnalyst performed less well, and DistillerSR offered no relative advantage. Research staff rated Abstrackr usable on the System Usability Scale. When the tools excluded records automatically, Abstrackr missed a median 5% of relevant records and up to 28%."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Abstrackr (machine learning citation screening)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"Assessors for AHRQ find that Abstrackr's predictions caught 33–90% of records that a single reviewer wrongly excluded in screening simulations","headline":"*Abstrackr* catches studies that a lone reviewer wrongly excluded"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["900c90069854f32cafcbdb5d9f31ef6a735d5a9cac5ddd23a43067ef677ead8a"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"The Agency for Healthcare Research and Quality (AHRQ) published a methods report by its University of Alberta Evidence-based Practice Center in November 2019. The authors tested three tools that use machine learning to screen records for systematic reviews. They ran retrospective simulations on three reviews. In each tool they screened a training set of 200 records, and the tool predicted the relevance of the rest. In the semi-automated simulation, the predictions complemented the work of a single reviewer. There, Abstrackr identified 33–90% of records that a single reviewer had wrongly excluded. A median 1% of relevant records were missed, and the workload fell by a median 40%. RobotAnalyst performed less well, and DistillerSR offered no relative advantage. Research staff rated Abstrackr usable on the System Usability Scale. When the tools excluded records automatically, Abstrackr missed a median 5% of relevant records and up to 28%.","replication":{"produced_by":["A senior reviewer screened a training set of 200 records in the tool.","The tool's relevance predictions for the remaining records were combined with a single reviewer's decisions."],"must_hold":["The labels in the training set are accurate.","A human reviewer still screens every record, since automatic exclusion missed up to 28% of relevant records with Abstrackr."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":28,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":26,"review_not_assessed":63,"not_assessed":0,"complete":false,"by_state":{"value_present":28,"not_reported":0,"unresolved":63,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"f709055a5303f5f030741d04c2fd4b594b8ebeb70ee048f1ddfc2a079b342bf2"}