{"id":"AIBD-0925","title":"Guidehouse, a reviewer proposed by Meta under its Justice Department settlement, verifies that the Variance Reduction System met housing-ad targets","summary":"Guidehouse verified that Meta met its housing-ad targets from January to April 2024. Of ads with at least 300 impressions, 94.2% had a sex variance of 10% or less, against a required 90.2%.","date_observed":"2024-01-01","date_basis":"event","date_precision":"year","benefit_status":"reported","source_collection":null,"source_entry":null,"account":"In June 2022 Meta settled a suit in which the U.S. Department of Justice alleged that its housing ads violated the Fair Housing Act. Meta denied liability. Under the settlement, Meta built the Variance Reduction System to bring each housing ad's audience closer to the eligible audience in sex and estimated race or ethnicity. Guidehouse, which Meta proposed as reviewer with the department's consent, checked Meta's compliance report for 1 January to 30 April 2024. In its report of 28 June 2024, Guidehouse states that it recomputed coverage from aggregated data that Meta supplied. Its figures differed from Meta's by 0.0%. Among ads with at least 300 impressions, 94.2% had a sex variance of 10% or less, against a required 90.2%. For estimated race or ethnicity the share was 81.4%, against 80.1%. Guidehouse found that Meta met every agreed metric. It tested Meta's sampling and race estimation on synthetic data, since disaggregated data were not available. It gives no figures for delivery without the system. On 30 October 2024 it verified compliance again for May to August 2024.","category":"aggregate result","unexpected":false,"purpose":"Meta built the Variance Reduction System to reduce the gap between the sex and estimated race or ethnicity mix of the people eligible to see a housing ad and of the people who see it.","system":"Variance Reduction System (VRS)","model_version":null,"developer":"Meta Platforms, Inc.","deployer":"Meta Platforms, Inc. (housing ads on Facebook, Instagram and Messenger)","location":"United States","domain":"Housing advertising","segment":null,"benefit_type":"Increased Inclusion","work_activities":["4.A.1.a.2","4.A.2.b.1"],"taxonomy_note":"The good is housing-ad delivery whose sex and estimated race or ethnicity mix stays close to that of the eligible audience, which the type names only broadly as acceptance of people with diverse backgrounds.","registry_ids":{},"ratings":[{"group":"societal","depth":null,"reason":"The reviewer gives no figure for delivery without the system, so the size of the reduction in skew cannot be established."}],"ai_role":{"status":"unassessed","statement":"Guidehouse recomputed coverage from aggregated data that Meta supplied and compared it with the agreed metrics. It did not compare delivery with and without the system. Its report defines the VRS only as a Meta-developed system designed to reduce variance and does not say that it learns from data. In a white paper of January 2023, Meta describes the system as an offline reinforcement learning framework."},"evidence":[{"kind":"evaluation report","citation":"Guidehouse Inc., independent third-party reviewer under the Settlement Agreement in United States v. Meta Platforms, Inc., No. 22-Civ-5187 (S.D.N.Y.). Meta VRS Compliance Metrics Verification, 28 June 2024. I. Executive Summary, p. 1, with Tables 1 and 2, p. 3. Meta proposed Guidehouse as reviewer with the consent of the Department of Justice. Published by the U.S. Department of Justice, Civil Rights Division.","title":"Meta VRS Compliance Metrics Verification","url":"https://www.justice.gov/crt/media/1362086/dl","date":"2024-06-28","written_by":"evaluator","quote":"Guidehouse reviewed the Meta Compliance Report dated May 30, 2024 for the reporting period from January 1, 2024 to April 30, 2024 (Reporting Period) and verified that Meta complied with the relevant VRS Compliance Metrics for both sex and estimated race / ethnicity"},{"kind":"evaluation report","citation":"Guidehouse Inc. Meta VRS Compliance Metrics Verification, 30 October 2024, for the reporting period of 1 May to 31 August 2024. I. Executive Summary, p. 1. Published by the U.S. Department of Justice, Civil Rights Division.","title":"Meta VRS Compliance Metrics Verification","url":"https://www.justice.gov/crt/media/1385866/dl","date":"2024-10-30","written_by":"evaluator","quote":"Guidehouse reviewed the Meta Compliance Report dated September 30, 2024 for the reporting period from May 1, 2024 to August 31, 2024 (Reporting Period) and verified that Meta complied with the relevant VRS Compliance Metrics for both sex and estimated race / ethnicity"},{"kind":"report","citation":"Meta. Toward fairness in personalized ads. White paper by Miranda Bogen and eight co-authors, January 2023. Section on ad delivery outcomes, How it works, p. 14.","title":"Toward fairness in personalized ads","url":"https://about.fb.com/wp-content/uploads/2023/01/Toward_fairness_in_personalized_ads.pdf","date":"2023-01-09","written_by":"makers","quote":"we formulate the Variance Reduction System as an offline reinforcement learning framework with the explicit goal of minimizing an ad's impression variance across the demographic subgroups","supports_benefit":false}],"replication":{"produced_by":["Meta ran housing ads through the VRS, which aims to reduce the variance between the eligible and actual audiences for sex and estimated race or ethnicity.","Meta estimated race or ethnicity with privacy-enhanced Bayesian Improved Surname Geocoding and measured variance with Earth Mover's Distance.","Meta measured each eligible audience from a sample of users, with a target of 6,000 and an expected minimum of 4,500."],"must_hold":["The sampling process must return large enough samples. An API issue left some samples too small from 1 to 15 January 2024.","Users' sex and ZIP codes must be known for their impressions to count in the calculation.","Housing ads tagged with other special ad categories must be counted, since Meta's calculation left some out."],"echoes":[]},"paired_incident":null,"headline":"*Variance Reduction System* keeps Meta housing ads within agreed limits","metric":"94.2% had a sex variance of 10% or less, against a required 90.2%","comparator":{"against":"The VRS Compliance Metrics that the Department of Justice and Meta agreed and the court adopted","before":"Required coverage of 90.2% (sex, variance of 10% or less, ads with at least 300 impressions)","after":"94.2%","basis":"measured","quote":"The difference in Coverage computed by Meta and computed by Guidehouse across all VRS Compliance Metrics was zero percent, and these figures were higher than the required VRS Compliance Metrics.","url":"https://www.justice.gov/crt/media/1362086/dl"},"later":{"status":"persisted","date":"2024-10-30","note":"Guidehouse verified that Meta again met the agreed metrics for May to August 2024.","quote":"Guidehouse reviewed the Meta Compliance Report dated September 30, 2024 for the reporting period from May 1, 2024 to August 31, 2024 (Reporting Period) and verified that Meta complied with the relevant VRS Compliance Metrics for both sex and estimated race / ethnicity","url":"https://www.justice.gov/crt/media/1385866/dl"},"value":null,"developer_country":"United States","deployer_country":"United States","unrated_because":"The reviewer measured delivery only with the system in place and against agreed targets, so the extent of the benefit to people seeking housing cannot be established.","brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of Guidehouse, the reviewer under the settlement between the U.S. Department of Justice and Meta.","source":"https://www.justice.gov/crt/media/1362086/dl","editor":"AIBD"}],"confidence":"claimed","shade":"low","claim_status":"The evaluator's account","claim_basis":"Sources: the makers' report and two evaluation reports from the evaluators. For the benefit itself, the cited sources include no independent source and no finding by an external body.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0925","catalog_version":"1.0.0","source_fingerprint":"8522554dde98553e37d4e8f76dde8a34e276cb191367662137358296fcecb9a7","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"},{"collection_id":"querist","version":"2026-10-25"}],"source_ids":["AIBD-0925-source-0001","AIBD-0925-source-0002","AIBD-0925-source-0003"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Meta Platforms, Inc.","deployer":"Meta Platforms, Inc. (housing ads on Facebook, Instagram and Messenger)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"aggregate result","ai_role":{"status":"unassessed","statement":"Guidehouse recomputed coverage from aggregated data that Meta supplied and compared it with the agreed metrics. It did not compare delivery with and without the system. Its report defines the VRS only as a Meta-developed system designed to reduce variance and does not say that it learns from data. In a white paper of January 2023, Meta describes the system as an offline reinforcement learning framework."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"low","evidence_word":"claimed","claim_status":"The evaluator's account","basis":"Sources: the makers' report and two evaluation reports from the evaluators. For the benefit itself, the cited sources include no independent source and no finding by an external body.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"societal","reason":"The reviewer gives no figure for delivery without the system, so the size of the reduction in skew cannot be established."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"societal","reason":"The reviewer gives no figure for delivery without the system, so the size of the reduction in skew cannot be established."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"The evaluator's account. Sources: the makers' report and two evaluation reports from the evaluators. For the benefit itself, the cited sources include no independent source and no finding by an external body.","evidence_word":"claimed","claim_status":"The evaluator's account"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0925-source-0001","AIBD-0925-source-0002","AIBD-0925-source-0003"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The evaluator's account. Sources: the makers' report and two evaluation reports from the evaluators. For the benefit itself, the cited sources include no independent source and no finding by an external body.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"routine use","answer":"The system was in routine use. Meta ran housing ads on its platforms through the system throughout the reporting period, as the settlement requires.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0925-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was in routine use. Meta ran housing ads on its platforms through the system throughout the reporting period, as the settlement requires.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"later":{"status":"persisted","date":"2024-10-30","note":"Guidehouse verified that Meta again met the agreed metrics for May to August 2024.","quote":"Guidehouse reviewed the Meta Compliance Report dated September 30, 2024 for the reporting period from May 1, 2024 to August 31, 2024 (Reporting Period) and verified that Meta complied with the relevant VRS Compliance Metrics for both sex and estimated race / ethnicity","url":"https://www.justice.gov/crt/media/1385866/dl"}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["later"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0925"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Increased Inclusion","taxonomy_note":"The good is housing-ad delivery whose sex and estimated race or ethnicity mix stays close to that of the eligible audience, which the type names only broadly as acceptance of people with diverse backgrounds."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type","taxonomy_note"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"In June 2022 Meta settled a suit in which the U.S. Department of Justice alleged that its housing ads violated the Fair Housing Act. Meta denied liability. Under the settlement, Meta built the Variance Reduction System to bring each housing ad's audience closer to the eligible audience in sex and estimated race or ethnicity. Guidehouse, which Meta proposed as reviewer with the department's consent, checked Meta's compliance report for 1 January to 30 April 2024. In its report of 28 June 2024, Guidehouse states that it recomputed coverage from aggregated data that Meta supplied. Its figures differed from Meta's by 0.0%. Among ads with at least 300 impressions, 94.2% had a sex variance of 10% or less, against a required 90.2%. For estimated race or ethnicity the share was 81.4%, against 80.1%. Guidehouse found that Meta met every agreed metric. It tested Meta's sampling and race estimation on synthetic data, since disaggregated data were not available. It gives no figures for delivery without the system. On 30 October 2024 it verified compliance again for May to August 2024.","benefit_type":"Increased Inclusion"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"societal","reason":"The reviewer gives no figure for delivery without the system, so the size of the reduction in skew cannot be established."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"reported"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"Meta built the Variance Reduction System to reduce the gap between the sex and estimated race or ethnicity mix of the people eligible to see a housing ad and of the people who see it."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"United States","developer_country":"United States","deployer_country":"United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"Meta built the Variance Reduction System to reduce the gap between the sex and estimated race or ethnicity mix of the people eligible to see a housing ad and of the people who see it."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2024-01-01","date_basis":"event","date_precision":"year","revisions":[{"date":"2026-10-31","change":"Added the record from the finding of Guidehouse, the reviewer under the settlement between the U.S. Department of Justice and Meta.","source":"https://www.justice.gov/crt/media/1362086/dl","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["Meta ran housing ads through the VRS, which aims to reduce the variance between the eligible and actual audiences for sex and estimated race or ethnicity.","Meta estimated race or ethnicity with privacy-enhanced Bayesian Improved Surname Geocoding and measured variance with Earth Mover's Distance.","Meta measured each eligible audience from a sample of users, with a target of 6,000 and an expected minimum of 4,500."],"must_hold":["The sampling process must return large enough samples. An API issue left some samples too small from 1 to 15 January 2024.","Users' sex and ZIP codes must be known for their impressions to count in the calculation.","Housing ads tagged with other special ad categories must be counted, since Meta's calculation left some out."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-31","change":"Added the record from the finding of Guidehouse, the reviewer under the settlement between the U.S. Department of Justice and Meta.","source":"https://www.justice.gov/crt/media/1362086/dl","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Housing advertising"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"evaluation report","citation":"Guidehouse Inc., independent third-party reviewer under the Settlement Agreement in United States v. Meta Platforms, Inc., No. 22-Civ-5187 (S.D.N.Y.). Meta VRS Compliance Metrics Verification, 28 June 2024. I. Executive Summary, p. 1, with Tables 1 and 2, p. 3. Meta proposed Guidehouse as reviewer with the consent of the Department of Justice. Published by the U.S. Department of Justice, Civil Rights Division.","title":"Meta VRS Compliance Metrics Verification","url":"https://www.justice.gov/crt/media/1362086/dl","date":"2024-06-28","written_by":"evaluator","quote":"Guidehouse reviewed the Meta Compliance Report dated May 30, 2024 for the reporting period from January 1, 2024 to April 30, 2024 (Reporting Period) and verified that Meta complied with the relevant VRS Compliance Metrics for both sex and estimated race / ethnicity"},{"kind":"evaluation report","citation":"Guidehouse Inc. Meta VRS Compliance Metrics Verification, 30 October 2024, for the reporting period of 1 May to 31 August 2024. I. Executive Summary, p. 1. Published by the U.S. Department of Justice, Civil Rights Division.","title":"Meta VRS Compliance Metrics Verification","url":"https://www.justice.gov/crt/media/1385866/dl","date":"2024-10-30","written_by":"evaluator","quote":"Guidehouse reviewed the Meta Compliance Report dated September 30, 2024 for the reporting period from May 1, 2024 to August 31, 2024 (Reporting Period) and verified that Meta complied with the relevant VRS Compliance Metrics for both sex and estimated race / ethnicity"},{"kind":"report","citation":"Meta. Toward fairness in personalized ads. White paper by Miranda Bogen and eight co-authors, January 2023. Section on ad delivery outcomes, How it works, p. 14.","title":"Toward fairness in personalized ads","url":"https://about.fb.com/wp-content/uploads/2023/01/Toward_fairness_in_personalized_ads.pdf","date":"2023-01-09","written_by":"makers","quote":"we formulate the Variance Reduction System as an offline reinforcement learning framework with the explicit goal of minimizing an ad's impression variance across the demographic subgroups","supports_benefit":false}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0925-source-0001","AIBD-0925-source-0002","AIBD-0925-source-0003"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"evaluation report","citation":"Guidehouse Inc., independent third-party reviewer under the Settlement Agreement in United States v. Meta Platforms, Inc., No. 22-Civ-5187 (S.D.N.Y.). Meta VRS Compliance Metrics Verification, 28 June 2024. I. Executive Summary, p. 1, with Tables 1 and 2, p. 3. Meta proposed Guidehouse as reviewer with the consent of the Department of Justice. Published by the U.S. Department of Justice, Civil Rights Division.","title":"Meta VRS Compliance Metrics Verification","url":"https://www.justice.gov/crt/media/1362086/dl","date":"2024-06-28","written_by":"evaluator","quote":"Guidehouse reviewed the Meta Compliance Report dated May 30, 2024 for the reporting period from January 1, 2024 to April 30, 2024 (Reporting Period) and verified that Meta complied with the relevant VRS Compliance Metrics for both sex and estimated race / ethnicity"},{"kind":"evaluation report","citation":"Guidehouse Inc. Meta VRS Compliance Metrics Verification, 30 October 2024, for the reporting period of 1 May to 31 August 2024. I. Executive Summary, p. 1. Published by the U.S. Department of Justice, Civil Rights Division.","title":"Meta VRS Compliance Metrics Verification","url":"https://www.justice.gov/crt/media/1385866/dl","date":"2024-10-30","written_by":"evaluator","quote":"Guidehouse reviewed the Meta Compliance Report dated September 30, 2024 for the reporting period from May 1, 2024 to August 31, 2024 (Reporting Period) and verified that Meta complied with the relevant VRS Compliance Metrics for both sex and estimated race / ethnicity"},{"kind":"report","citation":"Meta. Toward fairness in personalized ads. White paper by Miranda Bogen and eight co-authors, January 2023. Section on ad delivery outcomes, How it works, p. 14.","title":"Toward fairness in personalized ads","url":"https://about.fb.com/wp-content/uploads/2023/01/Toward_fairness_in_personalized_ads.pdf","date":"2023-01-09","written_by":"makers","quote":"we formulate the Variance Reduction System as an offline reinforcement learning framework with the explicit goal of minimizing an ad's impression variance across the demographic subgroups","supports_benefit":false}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"Guidehouse verified that Meta met its housing-ad targets from January to April 2024. Of ads with at least 300 impressions, 94.2% had a sex variance of 10% or less, against a required 90.2%.","account":"In June 2022 Meta settled a suit in which the U.S. Department of Justice alleged that its housing ads violated the Fair Housing Act. Meta denied liability. Under the settlement, Meta built the Variance Reduction System to bring each housing ad's audience closer to the eligible audience in sex and estimated race or ethnicity. Guidehouse, which Meta proposed as reviewer with the department's consent, checked Meta's compliance report for 1 January to 30 April 2024. In its report of 28 June 2024, Guidehouse states that it recomputed coverage from aggregated data that Meta supplied. Its figures differed from Meta's by 0.0%. Among ads with at least 300 impressions, 94.2% had a sex variance of 10% or less, against a required 90.2%. For estimated race or ethnicity the share was 81.4%, against 80.1%. Guidehouse found that Meta met every agreed metric. It tested Meta's sampling and race estimation on synthetic data, since disaggregated data were not available. It gives no figures for delivery without the system. On 30 October 2024 it verified compliance again for May to August 2024."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Variance Reduction System (VRS)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"Guidehouse, a reviewer proposed by Meta under its Justice Department settlement, verifies that the Variance Reduction System met housing-ad targets","headline":"*Variance Reduction System* keeps Meta housing ads within agreed limits"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["8fc86401e4a3c7786a8be4917ba0033b826e9e8af2ec7e98c204e70632cf529c"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"In June 2022 Meta settled a suit in which the U.S. Department of Justice alleged that its housing ads violated the Fair Housing Act. Meta denied liability. Under the settlement, Meta built the Variance Reduction System to bring each housing ad's audience closer to the eligible audience in sex and estimated race or ethnicity. Guidehouse, which Meta proposed as reviewer with the department's consent, checked Meta's compliance report for 1 January to 30 April 2024. In its report of 28 June 2024, Guidehouse states that it recomputed coverage from aggregated data that Meta supplied. Its figures differed from Meta's by 0.0%. Among ads with at least 300 impressions, 94.2% had a sex variance of 10% or less, against a required 90.2%. For estimated race or ethnicity the share was 81.4%, against 80.1%. Guidehouse found that Meta met every agreed metric. It tested Meta's sampling and race estimation on synthetic data, since disaggregated data were not available. It gives no figures for delivery without the system. On 30 October 2024 it verified compliance again for May to August 2024.","replication":{"produced_by":["Meta ran housing ads through the VRS, which aims to reduce the variance between the eligible and actual audiences for sex and estimated race or ethnicity.","Meta estimated race or ethnicity with privacy-enhanced Bayesian Improved Surname Geocoding and measured variance with Earth Mover's Distance.","Meta measured each eligible audience from a sample of users, with a target of 6,000 and an expected minimum of 4,500."],"must_hold":["The sampling process must return large enough samples. An API issue left some samples too small from 1 to 15 January 2024.","Users' sex and ZIP codes must be known for their impressions to count in the calculation.","Housing ads tagged with other special ad categories must be counted, since Meta's calculation left some out."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":28,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":26,"review_not_assessed":63,"not_assessed":0,"complete":false,"by_state":{"value_present":28,"not_reported":0,"unresolved":63,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"11087cd9d801cc5d328fe0baaa76b33c5090c0be7293f6b9845e617118fee2ff"}