{"id":"AIBD-0674","title":"Most staff surveyed by SSA's Inspector General found Insight useful in reviewing hearing decisions, though 71 reported problems with its flags","summary":"About 74% of 302 staff who answered a survey by SSA's Inspector General found Insight useful in reviewing hearing decisions, and 71 users reported problems with its flags.","date_observed":"2018-01-01","date_basis":"event","date_precision":"year","benefit_status":"mixed","source_collection":null,"source_entry":null,"account":"Staff in the Social Security Administration's Office of Appellate Operations built Insight. It uses natural language processing to flag possible errors of policy and consistency in decisions from disability hearings. The appellate office released it to its analysts in August 2017, and all hearing offices had it by September 2018. SSA's Office of the Inspector General reviewed its use from January to November 2018. It surveyed 633 staff with access to Insight and received 302 answers. About 74% agreed that it was useful and valuable when they reviewed decisions, and about 26% disagreed. Sixty-seven users wrote that it helped their reviews, and 71 thought that its flags had accuracy problems. In its report of 19 April 2019, the Inspector General also relays a study by the appellate office. Cases of Insight users were returned for quality problems 12.6% of the time, against 21.5% for non-users. The Inspector General did not test these figures. It found that SSA was not tracking whether Insight met its goals.","category":"aggregate result","unexpected":false,"purpose":"Staff in SSA's Office of Appellate Operations built Insight to flag possible errors of policy and internal consistency in decisions from disability hearings.","system":"Insight (SSA quality-review software for disability hearing decisions)","model_version":null,"developer":"Staff of SSA's Office of Appellate Operations (SSA's Office of Analytics and Improvements took over the program after its release)","deployer":"SSA Office of Appellate Operations and Office of Hearings Operations","location":"United States (national)","domain":"Social security (disability appeals)","segment":null,"benefit_type":"Improved Task Accuracy","work_activities":["4.A.2.a.3","4.A.2.a.4"],"taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"organizational","depth":1,"reason":"About 74% of the staff who answered the survey found Insight useful and valuable in reviewing decisions. The Inspector General could not determine whether it improved decisions in the hearing offices."}],"ai_role":{"status":"unassessed","statement":"The Inspector General's own evidence is a survey of staff opinion. The comparison of users with non-users comes from the appellate office's own study, which the Inspector General relays without testing. The Inspector General describes Insight as a program that uses natural language processing and artificial intelligence technologies."},"evidence":[{"kind":"audit report","citation":"Social Security Administration, Office of the Inspector General. The Social Security Administration's Use of Insight Software to Identify Potential Anomalies in Hearing Decisions. Audit report A-12-18-50353, 19 April 2019 (archived copy of the report at oig-files.ssa.gov).","title":"The Social Security Administration's Use of Insight Software to Identify Potential Anomalies in Hearing Decisions","url":"https://web.archive.org/web/20251010181932id_/https://oig-files.ssa.gov/audits/full/A-12-18-50353.pdf","date":"2019-04-19","written_by":"independent","quote":"In addition, about 74 percent agreed the program was useful and valuable when they were reviewing decisions, while about 26 percent disagreed (see Figure 2)."}],"replication":{"produced_by":["Insight extracted information from the text of each hearing decision and flagged possible quality issues.","Each flag linked users to the relevant policies so that they could check the issue."],"must_hold":["Users must check each flag against policy before acting on it.","The flags must be accurate and consistent with the agency's disability policies."],"echoes":[]},"paired_incident":null,"headline":"Most staff found *Insight* useful for reviewing hearing decisions","metric":"About 74% agreed that it was useful and valuable","comparator":{"against":"Cases of appellate analysts who did not use Insight, in the appellate office's own study","before":"21.5 percent of non-Insight users' cases returned for quality issues","after":"12.6 percent for Insight users' cases","basis":"measured","quote":"21.5 percent of non-Insight users’ cases were returned for quality issues by an AAJ or appeals officer for further work—compared to 12.6 percent for Insight users’ cases.","url":"https://web.archive.org/web/20251010181932id_/https://oig-files.ssa.gov/audits/full/A-12-18-50353.pdf"},"later":null,"value":null,"developer_country":"United States","deployer_country":"United States","unrated_because":null,"brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the SSA Office of the Inspector General.","source":"https://web.archive.org/web/20251010181932id_/https://oig-files.ssa.gov/audits/full/A-12-18-50353.pdf","editor":"AIBD"}],"confidence":"found","shade":"high","claim_status":"Found by an auditor","claim_basis":"Sources: one independent audit report. The cited evidence includes no independent study or report.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0674","catalog_version":"1.0.0","source_fingerprint":"01ea94468142c733926ef78cbdf30260a7022886d68d13e7b44defc8406ac8a4","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-0674-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Staff of SSA's Office of Appellate Operations (SSA's Office of Analytics and Improvements took over the program after its release)","deployer":"SSA Office of Appellate Operations and Office of Hearings Operations"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"aggregate result","ai_role":{"status":"unassessed","statement":"The Inspector General's own evidence is a survey of staff opinion. The comparison of users with non-users comes from the appellate office's own study, which the Inspector General relays without testing. The Inspector General describes Insight as a program that uses natural language processing and artificial intelligence technologies."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"About 74% of the staff who answered the survey found Insight useful and valuable in reviewing decisions. The Inspector General could not determine whether it improved decisions in the hearing offices."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"found","claim_status":"Found by an auditor","basis":"Sources: one independent audit report. The cited evidence includes no independent study or report.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"About 74% of the staff who answered the survey found Insight useful and valuable in reviewing decisions. The Inspector General could not determine whether it improved decisions in the hearing offices."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"About 74% of the staff who answered the survey found Insight useful and valuable in reviewing decisions. The Inspector General could not determine whether it improved decisions in the hearing offices."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Found by an auditor. Sources: one independent audit report. The cited evidence includes no independent study or report.","evidence_word":"found","claim_status":"Found by an auditor"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0674-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Found by an auditor. Sources: one independent audit report. The cited evidence includes no independent study or report.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"scaling","answer":"The system was being scaled up. Insight was in use across the appellate office and in all hearing offices by September 2018.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0674-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was being scaled up. Insight was in use across the appellate office and in all hearing offices by September 2018.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0674"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Staff in the Social Security Administration's Office of Appellate Operations built Insight. It uses natural language processing to flag possible errors of policy and consistency in decisions from disability hearings. The appellate office released it to its analysts in August 2017, and all hearing offices had it by September 2018. SSA's Office of the Inspector General reviewed its use from January to November 2018. It surveyed 633 staff with access to Insight and received 302 answers. About 74% agreed that it was useful and valuable when they reviewed decisions, and about 26% disagreed. Sixty-seven users wrote that it helped their reviews, and 71 thought that its flags had accuracy problems. In its report of 19 April 2019, the Inspector General also relays a study by the appellate office. Cases of Insight users were returned for quality problems 12.6% of the time, against 21.5% for non-users. The Inspector General did not test these figures. It found that SSA was not tracking whether Insight met its goals.","benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","depth":1,"reason":"About 74% of the staff who answered the survey found Insight useful and valuable in reviewing decisions. The Inspector General could not determine whether it improved decisions in the hearing offices."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"mixed"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"Staff in SSA's Office of Appellate Operations built Insight to flag possible errors of policy and internal consistency in decisions from disability hearings."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"United States (national)","developer_country":"United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"Staff in SSA's Office of Appellate Operations built Insight to flag possible errors of policy and internal consistency in decisions from disability hearings."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2018-01-01","date_basis":"event","date_precision":"year","revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the SSA Office of the Inspector General.","source":"https://web.archive.org/web/20251010181932id_/https://oig-files.ssa.gov/audits/full/A-12-18-50353.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["Insight extracted information from the text of each hearing decision and flagged possible quality issues.","Each flag linked users to the relevant policies so that they could check the issue."],"must_hold":["Users must check each flag against policy before acting on it.","The flags must be accurate and consistent with the agency's disability policies."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of the SSA Office of the Inspector General.","source":"https://web.archive.org/web/20251010181932id_/https://oig-files.ssa.gov/audits/full/A-12-18-50353.pdf","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Social security (disability appeals)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"audit report","citation":"Social Security Administration, Office of the Inspector General. The Social Security Administration's Use of Insight Software to Identify Potential Anomalies in Hearing Decisions. Audit report A-12-18-50353, 19 April 2019 (archived copy of the report at oig-files.ssa.gov).","title":"The Social Security Administration's Use of Insight Software to Identify Potential Anomalies in Hearing Decisions","url":"https://web.archive.org/web/20251010181932id_/https://oig-files.ssa.gov/audits/full/A-12-18-50353.pdf","date":"2019-04-19","written_by":"independent","quote":"In addition, about 74 percent agreed the program was useful and valuable when they were reviewing decisions, while about 26 percent disagreed (see Figure 2)."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0674-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"audit report","citation":"Social Security Administration, Office of the Inspector General. The Social Security Administration's Use of Insight Software to Identify Potential Anomalies in Hearing Decisions. Audit report A-12-18-50353, 19 April 2019 (archived copy of the report at oig-files.ssa.gov).","title":"The Social Security Administration's Use of Insight Software to Identify Potential Anomalies in Hearing Decisions","url":"https://web.archive.org/web/20251010181932id_/https://oig-files.ssa.gov/audits/full/A-12-18-50353.pdf","date":"2019-04-19","written_by":"independent","quote":"In addition, about 74 percent agreed the program was useful and valuable when they were reviewing decisions, while about 26 percent disagreed (see Figure 2)."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"About 74% of 302 staff who answered a survey by SSA's Inspector General found Insight useful in reviewing hearing decisions, and 71 users reported problems with its flags.","account":"Staff in the Social Security Administration's Office of Appellate Operations built Insight. It uses natural language processing to flag possible errors of policy and consistency in decisions from disability hearings. The appellate office released it to its analysts in August 2017, and all hearing offices had it by September 2018. SSA's Office of the Inspector General reviewed its use from January to November 2018. It surveyed 633 staff with access to Insight and received 302 answers. About 74% agreed that it was useful and valuable when they reviewed decisions, and about 26% disagreed. Sixty-seven users wrote that it helped their reviews, and 71 thought that its flags had accuracy problems. In its report of 19 April 2019, the Inspector General also relays a study by the appellate office. Cases of Insight users were returned for quality problems 12.6% of the time, against 21.5% for non-users. The Inspector General did not test these figures. It found that SSA was not tracking whether Insight met its goals."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Insight (SSA quality-review software for disability hearing decisions)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"Most staff surveyed by SSA's Inspector General found Insight useful in reviewing hearing decisions, though 71 reported problems with its flags","headline":"Most staff found *Insight* useful for reviewing hearing decisions"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["27f7b2f01ce0ae4b10a85426753d164c04aabecd5df2055183995678be21fff5"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Staff in the Social Security Administration's Office of Appellate Operations built Insight. It uses natural language processing to flag possible errors of policy and consistency in decisions from disability hearings. The appellate office released it to its analysts in August 2017, and all hearing offices had it by September 2018. SSA's Office of the Inspector General reviewed its use from January to November 2018. It surveyed 633 staff with access to Insight and received 302 answers. About 74% agreed that it was useful and valuable when they reviewed decisions, and about 26% disagreed. Sixty-seven users wrote that it helped their reviews, and 71 thought that its flags had accuracy problems. In its report of 19 April 2019, the Inspector General also relays a study by the appellate office. Cases of Insight users were returned for quality problems 12.6% of the time, against 21.5% for non-users. The Inspector General did not test these figures. It found that SSA was not tracking whether Insight met its goals.","replication":{"produced_by":["Insight extracted information from the text of each hearing decision and flagged possible quality issues.","Each flag linked users to the relevant policies so that they could check the issue."],"must_hold":["Users must check each flag against policy before acting on it.","The flags must be accurate and consistent with the agency's disability policies."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":28,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":26,"review_not_assessed":63,"not_assessed":0,"complete":false,"by_state":{"value_present":28,"not_reported":0,"unresolved":63,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"69058b71e91f320e2ddeb115a5031f46917e2e0734ee37280ff9ee353261f074"}