{"id":"AIBD-0764","title":"University of Geneva researchers find that GraphCast and Pangu-Weather forecast the 2021 North American winter storm better than ECMWF's HRES","summary":"In three case studies, GraphCast and Pangu-Weather forecast the February 2021 North American winter storm better than ECMWF's HRES. They did less well on two heatwaves.","date_observed":"2025-01-02","date_basis":"source publication","date_precision":"day","benefit_status":"mixed","source_collection":null,"source_entry":null,"account":"Researchers at the University of Geneva and the Helmholtz Centre for Environmental Research in Leipzig ran FourCastNet, Pangu-Weather and GraphCast from their released code. They compared the forecasts with ECMWF's high-resolution model, HRES, for three extreme events. The authors report that the AI models forecast the February 2021 North American winter storm far better. Near College Station, Texas, errors in wind chill for Pangu-Weather and GraphCast were much lower than for HRES, especially from 9 to 17 February. FourCastNet had the largest errors. For the 2021 Pacific Northwest heatwave, the AI models matched HRES locally but did worse when errors were pooled over space and time. For the 2023 humid heatwave in South Asia, they underestimated the highest danger levels over Bangladesh. Artificial Intelligence for the Earth Systems published the study on 2 January 2025. The findings rest on three cases. The AI forecasts were scored against ERA5, and HRES against its own analysis.","category":"aggregate result","unexpected":false,"purpose":"Google DeepMind built GraphCast, Huawei built Pangu-Weather and NVIDIA built FourCastNet to forecast global weather with deep learning models trained on reanalysis data.","system":"GraphCast, Pangu-Weather and FourCastNet","model_version":null,"developer":"Google DeepMind (GraphCast), Huawei (Pangu-Weather) and NVIDIA (FourCastNet)","deployer":null,"location":"Texas and the southern United States (2021 winter storm), the Pacific Northwest (2021 heatwave) and South Asia (2023 humid heatwave)","domain":"Weather forecasting / extreme events","segment":null,"benefit_type":"Improved Task Accuracy","work_activities":["4.A.1.b.3"],"taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"organizational","depth":null,"reason":"The authors compare forecasts of past events, and no forecaster or utility is reported to have used them."}],"ai_role":{"status":"assessed","statement":"The authors ran the AI models themselves and compared their forecasts with HRES for the same events, places and lead times."},"evidence":[{"kind":"peer-reviewed study","citation":"Pasche OC, Wider J, Zhang Z, Zscheischler J, Engelke S. Validating deep learning weather forecast models on recent high-impact extreme events. Artificial Intelligence for the Earth Systems 4(1) (2025).","title":"Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events","url":"https://journals.ametsoc.org/view/journals/aies/4/1/AIES-D-24-0033.1.xml","date":"2025-01-02","written_by":"independent","quote":"We find that ML weather prediction models locally achieve similar accuracy to HRES on the record-shattering Pacific Northwest heatwave but underperform when aggregated over space and time. However, they forecast the compound winter storm substantially better."}],"replication":{"produced_by":["The authors ran the pretrained models released by their developers.","Forecasts from all initialization times were compared at grid cells near affected cities."],"must_hold":["Health indices need variables such as surface humidity, which the AI models did not predict.","Single case studies should not be used alone to judge forecasting systems."],"echoes":[]},"paired_incident":null,"headline":"*GraphCast* and *Pangu-Weather* beat ECMWF on the 2021 winter storm","metric":null,"comparator":{"against":"ECMWF HRES forecasts for the same events and lead times","before":"HRES errors in wind chill near College Station, Texas","after":"Much lower errors for Pangu-Weather and GraphCast, especially from 9 to 17 February","basis":"measured","quote":"Errors for Pangu-Weather and GraphCast are substantially lower than for HRES, especially between 9 and 17 February, after the peak of the winter storm.","url":"https://journals.ametsoc.org/view/journals/aies/4/1/AIES-D-24-0033.1.xml"},"later":null,"value":null,"developer_country":"Multiple: United Kingdom, China, United States","deployer_country":null,"unrated_because":"The study rests on forecasts of three past events made for comparison, so no realized gain is reported.","brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-31","change":"Added the record from the report of the University of Geneva researchers.","source":"https://journals.ametsoc.org/view/journals/aies/4/1/AIES-D-24-0033.1.xml","editor":"AIBD"}],"confidence":"studied","shade":"high","claim_status":"Independently studied","claim_basis":"Sources: one independent peer-reviewed study. The cited evidence includes no finding by an external body.","case_jurisdictions":["United States"],"profile":{"case_id":"AIBD-0764","catalog_version":"1.0.0","source_fingerprint":"dcd7118cbd95d8738a567b7df7cc1987d1d2196885bf9895cc91080f1e8bad0c","collection_refs":[{"collection_id":"aibd","version":"2026-10-31"}],"source_ids":["AIBD-0764-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Google DeepMind (GraphCast), Huawei (Pangu-Weather) and NVIDIA (FourCastNet)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"value_present","value":{"kind":"record_fields","fields":{"ai_role":{"status":"assessed","statement":"The authors ran the AI models themselves and compared their forecasts with HRES for the same events, places and lead times."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"aggregate result","ai_role":{"status":"assessed","statement":"The authors ran the AI models themselves and compared their forecasts with HRES for the same events, places and lead times."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"high","evidence_word":"studied","claim_status":"Independently studied","basis":"Sources: one independent peer-reviewed study. The cited evidence includes no finding by an external body.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The authors compare forecasts of past events, and no forecaster or utility is reported to have used them."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The authors compare forecasts of past events, and no forecaster or utility is reported to have used them."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"Independently studied. Sources: one independent peer-reviewed study. The cited evidence includes no finding by an external body.","evidence_word":"studied","claim_status":"Independently studied"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0764-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Independently studied. Sources: one independent peer-reviewed study. The cited evidence includes no finding by an external body.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"laboratory test","answer":"The system was tested in a laboratory. The researchers ran the models on three past events to compare them with HRES.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0764-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was tested in a laboratory. The researchers ran the models on three past events to compare them with HRES.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].comparator":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].design":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].effect":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].exposure":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].id":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].measurement_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].metric":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].reproduction_materials":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].setting_sample":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"evaluations[].uncertainty_limitations":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"evaluations"},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0764"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Researchers at the University of Geneva and the Helmholtz Centre for Environmental Research in Leipzig ran FourCastNet, Pangu-Weather and GraphCast from their released code. They compared the forecasts with ECMWF's high-resolution model, HRES, for three extreme events. The authors report that the AI models forecast the February 2021 North American winter storm far better. Near College Station, Texas, errors in wind chill for Pangu-Weather and GraphCast were much lower than for HRES, especially from 9 to 17 February. FourCastNet had the largest errors. For the 2021 Pacific Northwest heatwave, the AI models matched HRES locally but did worse when errors were pooled over space and time. For the 2023 humid heatwave in South Asia, they underestimated the highest danger levels over Bangladesh. Artificial Intelligence for the Earth Systems published the study on 2 January 2025. The findings rest on three cases. The AI forecasts were scored against ERA5, and HRES against its own analysis.","benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The authors compare forecasts of past events, and no forecaster or utility is reported to have used them."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_status":"mixed"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"Google DeepMind built GraphCast, Huawei built Pangu-Weather and NVIDIA built FourCastNet to forecast global weather with deep learning models trained on reanalysis data."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"Texas and the southern United States (2021 winter storm), the Pacific Northwest (2021 heatwave) and South Asia (2023 humid heatwave)","developer_country":"Multiple: United Kingdom, China, United States"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"Google DeepMind built GraphCast, Huawei built Pangu-Weather and NVIDIA built FourCastNet to forecast global weather with deep learning models trained on reanalysis data."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2025-01-02","date_basis":"source publication","date_precision":"day","revisions":[{"date":"2026-10-31","change":"Added the record from the report of the University of Geneva researchers.","source":"https://journals.ametsoc.org/view/journals/aies/4/1/AIES-D-24-0033.1.xml","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["The authors ran the pretrained models released by their developers.","Forecasts from all initialization times were compared at grid cells near affected cities."],"must_hold":["Health indices need variables such as surface humidity, which the AI models did not predict.","Single case studies should not be used alone to judge forecasting systems."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-31","change":"Added the record from the report of the University of Geneva researchers.","source":"https://journals.ametsoc.org/view/journals/aies/4/1/AIES-D-24-0033.1.xml","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Weather forecasting / extreme events"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"peer-reviewed study","citation":"Pasche OC, Wider J, Zhang Z, Zscheischler J, Engelke S. Validating deep learning weather forecast models on recent high-impact extreme events. Artificial Intelligence for the Earth Systems 4(1) (2025).","title":"Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events","url":"https://journals.ametsoc.org/view/journals/aies/4/1/AIES-D-24-0033.1.xml","date":"2025-01-02","written_by":"independent","quote":"We find that ML weather prediction models locally achieve similar accuracy to HRES on the record-shattering Pacific Northwest heatwave but underperform when aggregated over space and time. However, they forecast the compound winter storm substantially better."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0764-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"peer-reviewed study","citation":"Pasche OC, Wider J, Zhang Z, Zscheischler J, Engelke S. Validating deep learning weather forecast models on recent high-impact extreme events. Artificial Intelligence for the Earth Systems 4(1) (2025).","title":"Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events","url":"https://journals.ametsoc.org/view/journals/aies/4/1/AIES-D-24-0033.1.xml","date":"2025-01-02","written_by":"independent","quote":"We find that ML weather prediction models locally achieve similar accuracy to HRES on the record-shattering Pacific Northwest heatwave but underperform when aggregated over space and time. However, they forecast the compound winter storm substantially better."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"In three case studies, GraphCast and Pangu-Weather forecast the February 2021 North American winter storm better than ECMWF's HRES. They did less well on two heatwaves.","account":"Researchers at the University of Geneva and the Helmholtz Centre for Environmental Research in Leipzig ran FourCastNet, Pangu-Weather and GraphCast from their released code. They compared the forecasts with ECMWF's high-resolution model, HRES, for three extreme events. The authors report that the AI models forecast the February 2021 North American winter storm far better. Near College Station, Texas, errors in wind chill for Pangu-Weather and GraphCast were much lower than for HRES, especially from 9 to 17 February. FourCastNet had the largest errors. For the 2021 Pacific Northwest heatwave, the AI models matched HRES locally but did worse when errors were pooled over space and time. For the 2023 humid heatwave in South Asia, they underestimated the highest danger levels over Bangladesh. Artificial Intelligence for the Earth Systems published the study on 2 January 2025. The findings rest on three cases. The AI forecasts were scored against ERA5, and HRES against its own analysis."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"GraphCast, Pangu-Weather and FourCastNet"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"University of Geneva researchers find that GraphCast and Pangu-Weather forecast the 2021 North American winter storm better than ECMWF's HRES","headline":"*GraphCast* and *Pangu-Weather* beat ECMWF on the 2021 winter storm"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["6dacf328e858b72fa74078f7d7f4980872e1aa7bdd07e2426cabb93c90a3935d"],"role":"SHA-256 of the complete public record and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public record, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Researchers at the University of Geneva and the Helmholtz Centre for Environmental Research in Leipzig ran FourCastNet, Pangu-Weather and GraphCast from their released code. They compared the forecasts with ECMWF's high-resolution model, HRES, for three extreme events. The authors report that the AI models forecast the February 2021 North American winter storm far better. Near College Station, Texas, errors in wind chill for Pangu-Weather and GraphCast were much lower than for HRES, especially from 9 to 17 February. FourCastNet had the largest errors. For the 2021 Pacific Northwest heatwave, the AI models matched HRES locally but did worse when errors were pooled over space and time. For the 2023 humid heatwave in South Asia, they underestimated the highest danger levels over Bangladesh. Artificial Intelligence for the Earth Systems published the study on 2 January 2025. The findings rest on three cases. The AI forecasts were scored against ERA5, and HRES against its own analysis.","replication":{"produced_by":["The authors ran the pretrained models released by their developers.","Forecasts from all initialization times were compared at grid cells near affected cities."],"must_hold":["Health indices need variables such as surface humidity, which the AI models did not predict.","Single case studies should not be used alone to judge forecasting systems."],"echoes":[]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":28,"reviewed":2,"reviewed_complete":2,"reviewed_partial":0,"record_mapped":26,"review_not_assessed":63,"not_assessed":0,"complete":false,"by_state":{"value_present":28,"not_reported":0,"unresolved":63,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"8397a7e48b1a4f0d2c2b55d1aae7757d027597b97f8b42626c950c9c02b310a2"}