{"id":"AIBD-0728","title":"Stanford evaluators find that after Allegheny's LASSO screening model and policy changes, screened-in children were less often removed from home","summary":"Stanford evaluators found that after Allegheny's LASSO screening model and policy changes, the 180-day home removal risk after a screen-in fell 2.7 points. Acceptance for services fell 8 points.","date_observed":"2020-01-01","date_basis":"event","date_precision":"year","benefit_status":"mixed","source_collection":null,"source_entry":null,"account":"In November 2018, Allegheny County replaced the first Allegheny Family Screening Tool with a version built on a LASSO model. The model was re-estimated in July 2019, and related policies changed in October 2019. Jeremy Goldhaber-Fiebert and Lea Prince of Stanford University compared about 20,000 children in general protective service episodes from October 2019 with about 31,000 children before any tool. Child protective service episodes, for which the tool was not used, served as a second comparison. In a report dated 26 March 2023, they found that the 180-day risk of removal from home after a screened-in episode fell by 2.7 percentage points. After child protective service episodes, removals fell by a smaller 1.5 points. Repeat episodes within 180 days fell after both screen-ins and screen-outs. Racial differences in short-term outcomes generally narrowed between white and Black children. The share of screened-in children accepted for services fell by 8 percentage points. The evaluators state that the tool's effects cannot be fully separated from policy changes and from COVID-19.","category":"aggregate result","unexpected":false,"purpose":"Developers working for Allegheny County's Department of Human Services built the LASSO version of the Allegheny Family Screening Tool to help call screeners decide which child welfare referrals to investigate.","system":"Allegheny Family Screening Tool, LASSO version (predictive risk model for child welfare hotline screening)","model_version":null,"developer":"Not named in the evaluation (built for the Allegheny County Department of Human Services)","deployer":"Allegheny County Department of Human Services, Office of Children, Youth and Families","location":"Allegheny County, Pennsylvania, United States","domain":"Child welfare","segment":null,"benefit_type":"Reduced Risk","work_activities":["4.A.2.a.1","4.A.2.b.1"],"taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"personal","depth":2,"reason":"Children in screened-in episodes were 2.7 percentage points less likely to be removed from home within 180 days."}],"ai_role":{"status":"unassessed","statement":"The evaluators compared periods before and after the tool, with child protective service episodes as a second comparison. They state that the tool's effects cannot be fully separated from policy changes and COVID-19. The report notes that the model was re-estimated with new weights in July 2019."},"evidence":[{"kind":"evaluation report","citation":"Goldhaber-Fiebert JD, Prince L. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County (Phase 2). Stanford University, 26 March 2023. Published by the Allegheny County Department of Human Services.","title":"Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County (Phase 2)","url":"https://analytics.alleghenycounty.us/wp-content/uploads/2024/05/23-ACDHS-19_FamilyScreeningToolUpdate_v2.pdf","date":"2023-03-26","written_by":"evaluator","quote":"The 180-day risk of home removal after an index GPS screen-in episodes decreased by 2.7 percentage points with the implementation of LASSO and surrounding policy and practice changes."},{"kind":"peer-reviewed study","citation":"Gerchick M, Jegede T, Shah T, Gutierrez A, Beiers S, Shemtov N, Xu K, Samant A, Horowitz A. The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23), pp. 1292-1310, June 2023. doi:10.1145/3593013.3594081 (full text on aclu.org).","title":"The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool","url":"https://www.aclu.org/the-devil-is-in-the-details-interrogating-values-embedded-in-the-allegheny-family-screening-tool","date":"2023-06-12","written_by":"independent","quote":"But when we measure the AFST in a manner more closely aligned with how it is deployed — using a maximum score policy or a policy of assigning a “protocol” to each family — we sometimes see a lower AUC, a higher false positive rate, and greater racial disparities in performance results","supports_benefit":false}],"replication":{"produced_by":["The highest-scoring referrals triggered a High-Risk Protocol that defaults to a screen-in unless a supervisor overrides it.","The model was re-estimated in July 2019 with new data elements and weights."],"must_hold":["Referrals are processed quickly enough for the score to use current data.","Supervisors review the protocol defaults."],"echoes":[]},"paired_incident":null,"headline":"Updated *Allegheny Family Screening Tool* precedes fewer home removals","metric":"fell by 2.7 percentage points","comparator":{"against":"General protective service episodes before any tool, January 2015 to July 2016","before":null,"after":"180-day risk of home removal after a screen-in 2.7 percentage points lower","basis":"measured","quote":"The 180-day risk of home removal after an index GPS screen-in episodes decreased by 2.7 percentage points with the implementation of LASSO and surrounding policy and practice changes.","url":"https://analytics.alleghenycounty.us/wp-content/uploads/2024/05/23-ACDHS-19_FamilyScreeningToolUpdate_v2.pdf"},"later":{"status":"challenged","date":"2023-06-12","note":"An audit by ACLU and HRDAG staff found that the tool, measured as the county uses it, had lower accuracy, more false positives and wider racial disparities than the county reported.","quote":"But when we measure the AFST in a manner more closely aligned with how it is deployed — using a maximum score policy or a policy of assigning a “protocol” to each family — we sometimes see a lower AUC, a higher false positive rate, and greater racial disparities in performance results","url":"https://www.aclu.org/the-devil-is-in-the-details-interrogating-values-embedded-in-the-allegheny-family-screening-tool"},"value":null,"developer_country":null,"deployer_country":"United States","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-31","change":"Added the record from the report of the Stanford evaluators.","source":"https://analytics.alleghenycounty.us/wp-content/uploads/2024/05/23-ACDHS-19_FamilyScreeningToolUpdate_v2.pdf","editor":"AIBD"},{"date":"2026-10-31","change":"Added an independent peer-reviewed study that questions the benefit.","source":null,"editor":"AIBD"},{"date":"2026-10-31","change":"Recorded the later audit that challenges the result.","source":null,"editor":"AIBD"}],"confidence":"claimed","shade":"low","claim_status":"The evaluator's account","claim_basis":"Sources: one independent peer-reviewed study and the evaluators' evaluation report. For the benefit itself, the cited sources include no independent source and no finding by an external body.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0728","catalog_version":"1.0.0","source_fingerprint":"cff92f35111d43fed2ef513c2d7087221c6399e2e6b4791e49b3ef44a5af6b1a","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-0728-source-0001","AIBD-0728-source-0002"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Not named in the evaluation (built for the Allegheny County Department of Human Services)","deployer":"Allegheny County Department of Human Services, Office of Children, Youth and Families"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"aggregate result","ai_role":{"status":"unassessed","statement":"The evaluators compared periods before and after the tool, with child protective service episodes as a second comparison. They state that the tool's effects cannot be fully separated from policy changes and COVID-19. The report notes that the model was re-estimated with new weights in July 2019."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Children in screened-in episodes were 2.7 percentage points less likely to be removed from home within 180 days."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"low","evidence_word":"claimed","claim_status":"The evaluator's account","basis":"Sources: one independent peer-reviewed study and the evaluators' evaluation report. For the benefit itself, the cited sources include no independent source and no finding by an external body.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Children in screened-in episodes were 2.7 percentage points less likely to be removed from home within 180 days."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Children in screened-in episodes were 2.7 percentage points less likely to be removed from home within 180 days."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"The evaluator's account. Sources: the evaluators' evaluation report. The cited evidence includes no independent source and no finding by an external body.","evidence_word":"claimed","claim_status":"The evaluator's account"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0728-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The evaluator's account. Sources: the evaluators' evaluation report. The cited evidence includes no independent source and no finding by an external body.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"routine use","answer":"The system was in routine use. Call screeners used the LASSO score on general protective service referrals from October 2019.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0728-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was in routine use. Call screeners used the LASSO score on general protective service referrals from October 2019.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"later":{"status":"challenged","date":"2023-06-12","note":"An audit by ACLU and HRDAG staff found that the tool, measured as the county uses it, had lower accuracy, more false positives and wider racial disparities than the county reported.","quote":"But when we measure the AFST in a manner more closely aligned with how it is deployed — using a maximum score policy or a policy of assigning a “protocol” to each family — we sometimes see a lower AUC, a higher false positive rate, and greater racial disparities in performance results","url":"https://www.aclu.org/the-devil-is-in-the-details-interrogating-values-embedded-in-the-allegheny-family-screening-tool"}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["later"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0728"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Reduced Risk"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"In November 2018, Allegheny County replaced the first Allegheny Family Screening Tool with a version built on a LASSO model. The model was re-estimated in July 2019, and related policies changed in October 2019. Jeremy Goldhaber-Fiebert and Lea Prince of Stanford University compared about 20,000 children in general protective service episodes from October 2019 with about 31,000 children before any tool. Child protective service episodes, for which the tool was not used, served as a second comparison. In a report dated 26 March 2023, they found that the 180-day risk of removal from home after a screened-in episode fell by 2.7 percentage points. After child protective service episodes, removals fell by a smaller 1.5 points. Repeat episodes within 180 days fell after both screen-ins and screen-outs. Racial differences in short-term outcomes generally narrowed between white and Black children. The share of screened-in children accepted for services fell by 8 percentage points. The evaluators state that the tool's effects cannot be fully separated from policy changes and from COVID-19.","benefit_type":"Reduced Risk"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"personal","depth":2,"reason":"Children in screened-in episodes were 2.7 percentage points less likely to be removed from home within 180 days."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"mixed"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"Developers working for Allegheny County's Department of Human Services built the LASSO version of the Allegheny Family Screening Tool to help call screeners decide which child welfare referrals to investigate."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"Allegheny County, Pennsylvania, United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"Developers working for Allegheny County's Department of Human Services built the LASSO version of the Allegheny Family Screening Tool to help call screeners decide which child welfare referrals to investigate."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2020-01-01","date_basis":"event","date_precision":"year","revisions":[{"date":"2026-10-31","change":"Added the record from the report of the Stanford evaluators.","source":"https://analytics.alleghenycounty.us/wp-content/uploads/2024/05/23-ACDHS-19_FamilyScreeningToolUpdate_v2.pdf","editor":"AIBD"},{"date":"2026-10-31","change":"Added an independent peer-reviewed study that questions the benefit.","editor":"AIBD"},{"date":"2026-10-31","change":"Recorded the later audit that challenges the result.","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["The highest-scoring referrals triggered a High-Risk Protocol that defaults to a screen-in unless a supervisor overrides it.","The model was re-estimated in July 2019 with new data elements and weights."],"must_hold":["Referrals are processed quickly enough for the score to use current data.","Supervisors review the protocol defaults."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-31","change":"Added the record from the report of the Stanford evaluators.","source":"https://analytics.alleghenycounty.us/wp-content/uploads/2024/05/23-ACDHS-19_FamilyScreeningToolUpdate_v2.pdf","editor":"AIBD"},{"date":"2026-10-31","change":"Added an independent peer-reviewed study that questions the benefit.","editor":"AIBD"},{"date":"2026-10-31","change":"Recorded the later audit that challenges the result.","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Child welfare"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"evaluation report","citation":"Goldhaber-Fiebert JD, Prince L. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County (Phase 2). Stanford University, 26 March 2023. Published by the Allegheny County Department of Human Services.","title":"Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County (Phase 2)","url":"https://analytics.alleghenycounty.us/wp-content/uploads/2024/05/23-ACDHS-19_FamilyScreeningToolUpdate_v2.pdf","date":"2023-03-26","written_by":"evaluator","quote":"The 180-day risk of home removal after an index GPS screen-in episodes decreased by 2.7 percentage points with the implementation of LASSO and surrounding policy and practice changes."},{"kind":"peer-reviewed study","citation":"Gerchick M, Jegede T, Shah T, Gutierrez A, Beiers S, Shemtov N, Xu K, Samant A, Horowitz A. The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23), pp. 1292-1310, June 2023. doi:10.1145/3593013.3594081 (full text on aclu.org).","title":"The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool","url":"https://www.aclu.org/the-devil-is-in-the-details-interrogating-values-embedded-in-the-allegheny-family-screening-tool","date":"2023-06-12","written_by":"independent","quote":"But when we measure the AFST in a manner more closely aligned with how it is deployed — using a maximum score policy or a policy of assigning a “protocol” to each family — we sometimes see a lower AUC, a higher false positive rate, and greater racial disparities in performance results","supports_benefit":false}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0728-source-0001","AIBD-0728-source-0002"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"evaluation report","citation":"Goldhaber-Fiebert JD, Prince L. Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County (Phase 2). Stanford University, 26 March 2023. Published by the Allegheny County Department of Human Services.","title":"Impact Evaluation of a Predictive Risk Modeling Tool for Allegheny County (Phase 2)","url":"https://analytics.alleghenycounty.us/wp-content/uploads/2024/05/23-ACDHS-19_FamilyScreeningToolUpdate_v2.pdf","date":"2023-03-26","written_by":"evaluator","quote":"The 180-day risk of home removal after an index GPS screen-in episodes decreased by 2.7 percentage points with the implementation of LASSO and surrounding policy and practice changes."},{"kind":"peer-reviewed study","citation":"Gerchick M, Jegede T, Shah T, Gutierrez A, Beiers S, Shemtov N, Xu K, Samant A, Horowitz A. The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23), pp. 1292-1310, June 2023. doi:10.1145/3593013.3594081 (full text on aclu.org).","title":"The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool","url":"https://www.aclu.org/the-devil-is-in-the-details-interrogating-values-embedded-in-the-allegheny-family-screening-tool","date":"2023-06-12","written_by":"independent","quote":"But when we measure the AFST in a manner more closely aligned with how it is deployed — using a maximum score policy or a policy of assigning a “protocol” to each family — we sometimes see a lower AUC, a higher false positive rate, and greater racial disparities in performance results","supports_benefit":false}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"Stanford evaluators found that after Allegheny's LASSO screening model and policy changes, the 180-day home removal risk after a screen-in fell 2.7 points. Acceptance for services fell 8 points.","account":"In November 2018, Allegheny County replaced the first Allegheny Family Screening Tool with a version built on a LASSO model. The model was re-estimated in July 2019, and related policies changed in October 2019. Jeremy Goldhaber-Fiebert and Lea Prince of Stanford University compared about 20,000 children in general protective service episodes from October 2019 with about 31,000 children before any tool. Child protective service episodes, for which the tool was not used, served as a second comparison. In a report dated 26 March 2023, they found that the 180-day risk of removal from home after a screened-in episode fell by 2.7 percentage points. After child protective service episodes, removals fell by a smaller 1.5 points. Repeat episodes within 180 days fell after both screen-ins and screen-outs. Racial differences in short-term outcomes generally narrowed between white and Black children. The share of screened-in children accepted for services fell by 8 percentage points. The evaluators state that the tool's effects cannot be fully separated from policy changes and from COVID-19."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Allegheny Family Screening Tool, LASSO version (predictive risk model for child welfare hotline screening)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"Stanford evaluators find that after Allegheny's LASSO screening model and policy changes, screened-in children were less often removed from home","headline":"Updated *Allegheny Family Screening Tool* precedes fewer home removals"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["6a27b6094c2e747fbb167999ccf480a8ad4a98addfe9cb9872f199d6967fc9a1"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"In November 2018, Allegheny County replaced the first Allegheny Family Screening Tool with a version built on a LASSO model. The model was re-estimated in July 2019, and related policies changed in October 2019. Jeremy Goldhaber-Fiebert and Lea Prince of Stanford University compared about 20,000 children in general protective service episodes from October 2019 with about 31,000 children before any tool. Child protective service episodes, for which the tool was not used, served as a second comparison. In a report dated 26 March 2023, they found that the 180-day risk of removal from home after a screened-in episode fell by 2.7 percentage points. After child protective service episodes, removals fell by a smaller 1.5 points. Repeat episodes within 180 days fell after both screen-ins and screen-outs. Racial differences in short-term outcomes generally narrowed between white and Black children. The share of screened-in children accepted for services fell by 8 percentage points. The evaluators state that the tool's effects cannot be fully separated from policy changes and from COVID-19.","replication":{"produced_by":["The highest-scoring referrals triggered a High-Risk Protocol that defaults to a screen-in unless a supervisor overrides it.","The model was re-estimated in July 2019 with new data elements and weights."],"must_hold":["Referrals are processed quickly enough for the score to use current data.","Supervisors review the protocol defaults."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":29,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":27,"review_not_assessed":62,"not_assessed":0,"complete":false,"by_state":{"value_present":29,"not_reported":0,"unresolved":62,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"675711f01bef259d8920c8eacb9ada18892f0ef3b8ea6f27dc18b3f842cf186d"}