{"id":"AIBD-0657","title":"ECMWF's operational-like test finds Pangu-Weather forecasts about as accurate as its own IFS model","summary":"ECMWF ran Huawei's Pangu-Weather from its own analyses and found accuracy comparable to its IFS model, with better summer temperature-event forecasts in Europe.","date_observed":"2024-06-06","date_basis":"source publication","date_precision":"day","benefit_status":"intended","source_collection":null,"source_entry":null,"account":"Scientists at the European Centre for Medium-Range Weather Forecasts (ECMWF) set up Pangu-Weather, a machine-learning model that Huawei Cloud trained on ERA5 reanalysis data, on ECMWF computers. They started it from the same operational analyses as ECMWF's physics-based Integrated Forecasting System (IFS) and compared ten-day forecasts, mainly for the year from 1 March 2022 to 28 February 2023, with 2018 used for tropical cyclones. The authors report that accuracy was comparable for global scores and for extreme events, checked against IFS analyses and against station observations. For 2-meter temperature events over Europe in summer, Pangu-Weather discriminated events better than the IFS, with significant differences for thresholds between the 75th and 90th percentiles; in winter the two models performed alike. Differences in tropical cyclone track error were small and not significant. The authors also report weaknesses. The forecasts are too smooth, bias grows with lead time, and cyclone intensity is forecast poorly. The Bulletin of the American Meteorological Society published the study on 6 June 2024. The authors evaluated deterministic forecasts from one model and did not measure any use in warnings.","category":"general benefit","unexpected":false,"purpose":"Pangu-Weather was built to produce global medium-range weather forecasts at a small fraction of the computing cost of physics-based models. Forecast accuracy in an operational setting is that purpose.","system":"Pangu-Weather (PanguWeather, PGW)","model_version":null,"developer":"Huawei Cloud","deployer":"European Centre for Medium-Range Weather Forecasts (ECMWF), in an operational-like test","location":"ECMWF, Reading, United Kingdom, and Bonn, Germany; verification over the Northern Hemisphere and Europe","domain":"Weather forecasting","segment":null,"benefit_type":"Improved Task Accuracy","taxonomy_note":null,"registry_ids":{},"ratings":[{"group":"organizational","depth":null,"reason":"The study compares forecast skill on past periods and reports no realized gain for ECMWF or for any user of the forecasts."}],"ai_role":{"status":"assessed","statement":"ECMWF started Pangu-Weather and the IFS from the same operational analyses and verified both with the same tools, so the authors isolate the skill of the machine-learning model."},"evidence":[{"kind":"peer-reviewed study","citation":"Ben Bouallègue Z, Clare MCA, Magnusson L, et al. The rise of data-driven weather forecasting: a first statistical assessment of machine learning-based weather forecasts in an operational-like context. Bull Am Meteorol Soc. 2024;105(6):E864-E883.","title":"The Rise of Data-Driven Weather Forecasting: A First Statistical Assessment of Machine Learning-Based Weather Forecasts in an Operational-Like Context","url":"https://journals.ametsoc.org/view/journals/bams/105/6/BAMS-D-23-0162.1.xml","date":"2024-06-06","written_by":"evaluator","quote":"The results are very promising, with comparable accuracy for both global metrics and extreme events, when verified against both the operational IFS analysis and synoptic observations."}],"replication":{"produced_by":["ECMWF ran Pangu-Weather from its own operational IFS analyses.","Both models were verified with the same scores against analyses and station observations."],"must_hold":["The model needs high-quality initial conditions from a data assimilation system.","The weather to be forecast must resemble the reanalysis period used for training."],"echoes":[{"date":"2025-10-20","setting":"CIRA/Colorado State University operations-based verification of GraphCast, Pangu-Weather and other AI models for 2023 tropical cyclones (AIBD-0090)","model_version":null,"outcome":"recurred","source":"https://doi.org/10.1175/AIES-D-24-0085.1"}]},"paired_incident":null,"headline":"Europe's forecast center tests *Pangu-Weather* beside its own model","metric":"Pangu-Weather discriminated events better than the IFS","comparator":{"against":"ECMWF's operational IFS forecasts started from the same analyses","before":"IFS discrimination of summer 2-meter temperature events over Europe","after":"Significantly better discrimination by Pangu-Weather for thresholds between the 75th and 90th percentiles","basis":"measured","quote":"The IFS and PGW have similar levels of performance in winter, while PGW outperforms the IFS in summer with differences statistically significant for percentiles between 75% and 90%, as estimated by block-bootstrapping.","url":"https://journals.ametsoc.org/view/journals/bams/105/6/BAMS-D-23-0162.1.xml"},"later":null,"value":null,"developer_country":"China","deployer_country":"United Kingdom","unrated_because":"The study is a retrospective test of forecast skill, and no source reports that the forecasts were used or that anyone gained from them.","brief_notes":null,"brief":null,"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://journals.ametsoc.org/view/journals/bams/105/6/BAMS-D-23-0162.1.xml","editor":"AIBD"},{"date":"2026-10-29","change":"Corrected the benefit status; revised the depth ratings.","source":null,"editor":"AIBD"}],"confidence":"claimed","shade":"low","claim_status":"The evaluator's account","claim_basis":"Sources: the evaluators' peer-reviewed study. The cited evidence includes no independent source and no finding by a regulator or court.","case_jurisdictions":["United Kingdom","Germany"],"profile":{"case_id":"AIBD-0657","catalog_version":"1.0.0","source_fingerprint":"fc8898138d5635733920740a870892577920fe12e72664f0a052f69b26aba171","collection_refs":[{"collection_id":"aibd","version":"2026-10-29"},{"collection_id":"viator","version":"2026-10-25"}],"source_ids":["AIBD-0657-source-0001"],"families":{"strategic assessments":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"evaluations":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0657-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":[],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"partial","record_ids":["AIBD-0657-evaluation-1"]},"monetary observations":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]},"institutional decisions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"Editorial assignment.","review_state":"not_assessed","coverage":"partial","record_ids":[]}},"fields":{"actors[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"developer":"Huawei Cloud","deployer":"European Centre for Medium-Range Weather Forecasts (ECMWF), in an operational-like test"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["developer","deployer"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"actors[].ownership_control":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"ai_contribution":{"state":"value_present","value":{"kind":"record_fields","fields":{"ai_role":{"status":"assessed","statement":"ECMWF started Pangu-Weather and the IFS from the same operational analyses and verified both with the same tools, so the authors isolate the skill of the machine-learning model."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"ai_relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"category":"general benefit","ai_role":{"status":"assessed","statement":"ECMWF started Pangu-Weather and the IFS from the same operational analyses and verified both with the same tools, so the authors isolate the skill of the machine-learning model."}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ai_role","category"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].alternatives":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].assessor_date":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].depth":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"The record revision removed the mapped value.","record_fields":["ratings"],"source_refs":[]},"reason":"The revised record no longer states this information.","review_state":"not_assessed","coverage":"partial"},"assessments[].display_confidence":{"state":"value_present","value":{"kind":"structured","data":{"confidence":"low","evidence_word":"claimed","claim_status":"The evaluator's account","basis":"Sources: the evaluators' peer-reviewed study. The cited evidence includes no independent source and no finding by a regulator or court.","interpretation":"Existing display rule preserved; the richer outcome-specific confidence question still needs review."}},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rationale":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The study compares forecast skill on past periods and reports no realized gain for ECMWF or for any user of the forecasts."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"assessments[].rubric_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"assessments[].status_basis":{"state":"value_present","value":{"kind":"text","text":"Intended. ECMWF scored Pangu-Weather forecasts for past periods against its own model and against observations, which is a retrodiction, and the authors did not measure any use of the forecasts in operations or warnings."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Intended. ECMWF scored Pangu-Weather forecasts for past periods against its own model and against observations, which is a retrodiction, and the authors did not measure any use of the forecasts in operations or warnings.","inferred":false}},"assessments[].target":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The study compares forecast skill on past periods and reports no realized gain for ECMWF or for any user of the forecasts."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"availability":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"brief":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"case_kind":{"state":"value_present","value":{"kind":"structured","data":{"classification":"evaluation or study","answer":"The case describes an evaluation or a study. A verification study by a forecast center independent of the developer.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The case describes an evaluation or a study. A verification study by a forecast center independent of the developer.","inferred":false}},"case_links[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Compared with the incidents in the AI Incident Database snapshot of 26 October 2026.","record_fields":["paired_incident"],"source_refs":[]},"reason":"No incident in the AI Incident Database snapshot of 26 October 2026 concerns the same kind of system harming the same kind of stakeholder.","review_state":"reviewed","coverage":"partial"},"claims[].evidence_links":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].evidence_strength":{"state":"value_present","value":{"kind":"structured","data":{"answer":"The evaluator's account. Sources: the evaluators' peer-reviewed study. The cited evidence includes no independent source and no finding by a regulator or court.","evidence_word":"claimed","claim_status":"The evaluator's account"}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The evaluator's account. Sources: the evaluators' peer-reviewed study. The cited evidence includes no independent source and no finding by a regulator or court.","inferred":false}},"claims[].missingness":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"claims[].proposition":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"cross_border_flows[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"decisions[].action_outcome":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].grounds":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].institution_task":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"decisions[].success_criterion":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"institutional decisions"},"deployment_stage":{"state":"value_present","value":{"kind":"structured","data":{"classification":"laboratory test","answer":"The system was tested in a laboratory. ECMWF ran the model in an operational-like setting for research; it did not issue forecasts from it.","scope":"The period and activity described in the case."}},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["account","evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"The system was tested in a laboratory. ECMWF ran the model in an operational-like setting for research; it did not issue forecasts from it.","inferred":false}},"economics[].amount":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].costs[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].counterfactual":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].currency_price_basis":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].gross_net":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].measure_type":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].period_unit":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].realization":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].recipient_payer":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].return_payback":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"economics[].valuation_method":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"monetary observations"},"evaluations[].ai_uplift":{"state":"value_present","value":{"kind":"text","text":"Comparable accuracy at much lower computing cost; cost not quantified in the paper."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Comparable accuracy at much lower computing cost; cost not quantified in the paper.","inferred":false}},"evaluations[].comparator":{"state":"value_present","value":{"kind":"text","text":"Operational IFS forecasts from the same initial conditions."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Operational IFS forecasts from the same initial conditions.","inferred":false}},"evaluations[].design":{"state":"value_present","value":{"kind":"text","text":"Controlled observational."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Controlled observational.","inferred":false}},"evaluations[].effect":{"state":"value_present","value":{"kind":"text","text":"Comparable global skill; better summer event discrimination for the 75th to 90th percentiles; similar cyclone track error; worse cyclone intensity."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Comparable global skill; better summer event discrimination for the 75th to 90th percentiles; similar cyclone track error; worse cyclone intensity.","inferred":false}},"evaluations[].exposure":{"state":"value_present","value":{"kind":"text","text":"Research runs only."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Research runs only.","inferred":false}},"evaluations[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0657-evaluation-1"],"role":"Evaluations of the benefit"},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Controlled observational.","inferred":false}},"evaluations[].measurement_method":{"state":"value_present","value":{"kind":"text","text":"Standard verification scores (correlation, RMSE, area under the ROC curve, cyclone position and pressure error) against IFS analyses, synoptic observations and IBTrACS."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Standard verification scores (correlation, RMSE, area under the ROC curve, cyclone position and pressure error) against IFS analyses, synoptic observations and IBTrACS.","inferred":false}},"evaluations[].metric":{"state":"value_present","value":{"kind":"text","text":"Upper-air and 2-meter temperature skill; event discrimination; cyclone track and intensity error."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Upper-air and 2-meter temperature skill; event discrimination; cyclone track and intensity error.","inferred":false}},"evaluations[].reproduction_materials":{"state":"value_present","value":{"kind":"text","text":"Model weights public for noncommercial use; ECMWF toolkit public; ECMWF forecasts available."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Model weights public for noncommercial use; ECMWF toolkit public; ECMWF forecasts available.","inferred":false}},"evaluations[].setting_sample":{"state":"value_present","value":{"kind":"text","text":"Ten-day forecasts from 1 March 2022 to 28 February 2023, with 2018 for tropical cyclones; Northern Hemisphere and Europe."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Ten-day forecasts from 1 March 2022 to 28 February 2023, with 2018 for tropical cyclones; Northern Hemisphere and Europe.","inferred":false}},"evaluations[].uncertainty_limitations":{"state":"value_present","value":{"kind":"text","text":"Deterministic forecasts from one model; smooth fields and growing bias; poor cyclone intensity; no measure of use by forecasters."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Deterministic forecasts from one model; smooth fields and growing bias; poor cyclone intensity; no measure of use by forecasters.","inferred":false}},"external_references[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"follow_up[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"human_oversight":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"id":{"state":"value_present","value":{"kind":"record_fields","fields":{"id":"AIBD-0657"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["id"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"infrastructure_dependencies[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiaries":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].beneficiary_count":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].benefit_type":{"state":"value_present","value":{"kind":"record_fields","fields":{"benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].claim":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Scientists at the European Centre for Medium-Range Weather Forecasts (ECMWF) set up Pangu-Weather, a machine-learning model that Huawei Cloud trained on ERA5 reanalysis data, on ECMWF computers. They started it from the same operational analyses as ECMWF's physics-based Integrated Forecasting System (IFS) and compared ten-day forecasts, mainly for the year from 1 March 2022 to 28 February 2023, with 2018 used for tropical cyclones. The authors report that accuracy was comparable for global scores and for extreme events, checked against IFS analyses and against station observations. For 2-meter temperature events over Europe in summer, Pangu-Weather discriminated events better than the IFS, with significant differences for thresholds between the 75th and 90th percentiles; in winter the two models performed alike. Differences in tropical cyclone track error were small and not significant. The authors also report weaknesses. The forecasts are too smooth, bias grows with lead time, and cyclone intensity is forecast poorly. The Bulletin of the American Meteorological Society published the study on 6 June 2024. The authors evaluated deterministic forecasts from one model and did not measure any use in warnings.","benefit_type":"Improved Task Accuracy"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","benefit_type"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].distribution":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].duration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].id":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].stakeholder_group":{"state":"value_present","value":{"kind":"record_fields","fields":{"ratings":[{"group":"organizational","reason":"The study compares forecast skill on past periods and reports no realized gain for ECMWF or for any user of the forecasts."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["ratings"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"outcomes[].status":{"state":"value_present","value":{"kind":"text","text":"Intended. ECMWF scored Pangu-Weather forecasts for past periods against its own model and against observations, which is a retrodiction, and the authors did not measure any use of the forecasts in operations or warnings."},"provenance":{"kind":"source_review","basis":"Editorial assignment.","record_fields":["benefit_status"],"source_refs":["AIBD-0657-source-0001"],"reviewed_on":"2026-10-29"},"review_state":"reviewed","coverage":"complete","display":{"answer":"Intended. ECMWF scored Pangu-Weather forecasts for past periods against its own model and against observations, which is a retrodiction, and the authors did not measure any use of the forecasts in operations or warnings.","inferred":false}},"outcomes[].tradeoffs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"outcomes[].unexpected":{"state":"value_present","value":{"kind":"record_fields","fields":{"unexpected":false,"purpose":"Pangu-Weather was built to produce global medium-range weather forecasts at a small fraction of the computing cost of physics-based models. Forecast accuracy in an operational setting is that purpose."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["unexpected","purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"places[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"location":"ECMWF, Reading, United Kingdom, and Bonn, Germany; verification over the Northern Hemisphere and Europe","developer_country":"China","deployer_country":"United Kingdom"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["location","developer_country","deployer_country"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"purpose":{"state":"value_present","value":{"kind":"record_fields","fields":{"purpose":"Pangu-Weather was built to produce global medium-range weather forecasts at a small fraction of the computing cost of physics-based models. Forecast accuracy in an operational setting is that purpose."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["purpose"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"record_dates":{"state":"value_present","value":{"kind":"record_fields","fields":{"date_observed":"2024-06-06","date_basis":"source publication","date_precision":"day","revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://journals.ametsoc.org/view/journals/bams/105/6/BAMS-D-23-0162.1.xml","editor":"AIBD"},{"date":"2026-10-29","change":"Corrected the benefit status; revised the depth ratings.","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["date_observed","date_basis","date_precision","revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.attempts[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["ECMWF ran Pangu-Weather from its own operational IFS analyses.","Both models were verified with the same scores against analyses and station observations."],"must_hold":["The model needs high-quality initial conditions from a data assimilation system.","The weather to be forecast must resemble the reanalysis period used for training."],"echoes":[{"date":"2025-10-20","setting":"CIRA/Colorado State University operations-based verification of GraphCast, Pangu-Weather and other AI models for 2023 tropical cyclones (AIBD-0090)","outcome":"recurred","source":"https://doi.org/10.1175/AIES-D-24-0085.1"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"replication.conditions":{"state":"value_present","value":{"kind":"record_fields","fields":{"replication":{"produced_by":["ECMWF ran Pangu-Weather from its own operational IFS analyses.","Both models were verified with the same scores against analyses and station observations."],"must_hold":["The model needs high-quality initial conditions from a data assimilation system.","The weather to be forecast must resemble the reanalysis period used for training."],"echoes":[{"date":"2025-10-20","setting":"CIRA/Colorado State University operations-based verification of GraphCast, Pangu-Weather and other AI models for 2023 tropical cyclones (AIBD-0090)","outcome":"recurred","source":"https://doi.org/10.1175/AIES-D-24-0085.1"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"revision_history[]":{"state":"value_present","value":{"kind":"record_fields","fields":{"revisions":[{"date":"2026-10-29","change":"Added the case.","source":"https://journals.ametsoc.org/view/journals/bams/105/6/BAMS-D-23-0162.1.xml","editor":"AIBD"},{"date":"2026-10-29","change":"Corrected the benefit status; revised the depth ratings.","editor":"AIBD"}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["revisions"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sector":{"state":"value_present","value":{"kind":"record_fields","fields":{"domain":"Weather forecasting"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["domain"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].access_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].citation_locator":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"peer-reviewed study","citation":"Ben Bouallègue Z, Clare MCA, Magnusson L, et al. The rise of data-driven weather forecasting: a first statistical assessment of machine learning-based weather forecasts in an operational-like context. Bull Am Meteorol Soc. 2024;105(6):E864-E883.","title":"The Rise of Data-Driven Weather Forecasting: A First Statistical Assessment of Machine Learning-Based Weather Forecasts in an Operational-Like Context","url":"https://journals.ametsoc.org/view/journals/bams/105/6/BAMS-D-23-0162.1.xml","date":"2024-06-06","written_by":"evaluator","quote":"The results are very promising, with comparable accuracy for both global metrics and extreme events, when verified against both the operational IFS analysis and synoptic observations."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].id":{"state":"value_present","value":{"kind":"identifiers","identifiers":["AIBD-0657-source-0001"],"role":"Existing evidence items, in their original order"},"provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"sources[].language":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"sources[].relationship":{"state":"value_present","value":{"kind":"record_fields","fields":{"evidence":[{"kind":"peer-reviewed study","citation":"Ben Bouallègue Z, Clare MCA, Magnusson L, et al. The rise of data-driven weather forecasting: a first statistical assessment of machine learning-based weather forecasts in an operational-like context. Bull Am Meteorol Soc. 2024;105(6):E864-E883.","title":"The Rise of Data-Driven Weather Forecasting: A First Statistical Assessment of Machine Learning-Based Weather Forecasts in an Operational-Like Context","url":"https://journals.ametsoc.org/view/journals/bams/105/6/BAMS-D-23-0162.1.xml","date":"2024-06-06","written_by":"evaluator","quote":"The results are very promising, with comparable accuracy for both global metrics and extreme events, when verified against both the operational IFS analysis and synoptic observations."}]}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["evidence"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"sources[].upstream_sources":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"strategic_assessments[]":{"state":"unresolved","review_state":"not_assessed","coverage":"partial","family":"strategic assessments"},"summary":{"state":"value_present","value":{"kind":"record_fields","fields":{"summary":"ECMWF ran Huawei's Pangu-Weather from its own analyses and found accuracy comparable to its IFS model, with better summer temperature-event forecasts in Europe.","account":"Scientists at the European Centre for Medium-Range Weather Forecasts (ECMWF) set up Pangu-Weather, a machine-learning model that Huawei Cloud trained on ERA5 reanalysis data, on ECMWF computers. They started it from the same operational analyses as ECMWF's physics-based Integrated Forecasting System (IFS) and compared ten-day forecasts, mainly for the year from 1 March 2022 to 28 February 2023, with 2018 used for tropical cyclones. The authors report that accuracy was comparable for global scores and for extreme events, checked against IFS analyses and against station observations. For 2-meter temperature events over Europe in summer, Pangu-Weather discriminated events better than the IFS, with significant differences for thresholds between the 75th and 90th percentiles; in winter the two models performed alike. Differences in tropical cyclone track error were small and not significant. The authors also report weaknesses. The forecasts are too smooth, bias grows with lead time, and cyclone intensity is forecast poorly. The Bulletin of the American Meteorological Society published the study on 6 June 2024. The authors evaluated deterministic forecasts from one model and did not measure any use in warnings."}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["summary","account"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].autonomy_permissions":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].component_role":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].deployment_configuration":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].inputs_outputs":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_name":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].model_version":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"systems[].product_name":{"state":"value_present","value":{"kind":"record_fields","fields":{"system":"Pangu-Weather (PanguWeather, PGW)"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["system"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"systems[].provider":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"taxonomy_crosswalks[]":{"state":"unresolved","provenance":{"kind":"curatorial","basis":"Editorial assignment.","record_fields":[],"source_refs":[]},"reason":"This is an explicit unresolved assignment: the sources on record do not settle this question.","review_state":"not_assessed","coverage":"partial"},"title":{"state":"value_present","value":{"kind":"record_fields","fields":{"title":"ECMWF's operational-like test finds Pangu-Weather forecasts about as accurate as its own IFS model","headline":"Europe's forecast center tests *Pangu-Weather* beside its own model"}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["title","headline"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"},"version":{"state":"value_present","value":{"kind":"identifiers","identifiers":["31243383b1b5935071f805490f06aaefe1a92408a865ebfce8f99fc5fbb69314"],"role":"SHA-256 of the complete public case and its assessments"},"provenance":{"kind":"curatorial","basis":"SHA-256 of the public case, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.","record_fields":[],"source_refs":[]},"review_state":"record_mapped","coverage":"complete"},"workflow":{"state":"value_present","value":{"kind":"record_fields","fields":{"account":"Scientists at the European Centre for Medium-Range Weather Forecasts (ECMWF) set up Pangu-Weather, a machine-learning model that Huawei Cloud trained on ERA5 reanalysis data, on ECMWF computers. They started it from the same operational analyses as ECMWF's physics-based Integrated Forecasting System (IFS) and compared ten-day forecasts, mainly for the year from 1 March 2022 to 28 February 2023, with 2018 used for tropical cyclones. The authors report that accuracy was comparable for global scores and for extreme events, checked against IFS analyses and against station observations. For 2-meter temperature events over Europe in summer, Pangu-Weather discriminated events better than the IFS, with significant differences for thresholds between the 75th and 90th percentiles; in winter the two models performed alike. Differences in tropical cyclone track error were small and not significant. The authors also report weaknesses. The forecasts are too smooth, bias grows with lead time, and cyclone intensity is forecast poorly. The Bulletin of the American Meteorological Society published the study on 6 June 2024. The authors evaluated deterministic forecasts from one model and did not measure any use in warnings.","replication":{"produced_by":["ECMWF ran Pangu-Weather from its own operational IFS analyses.","Both models were verified with the same scores against analyses and station observations."],"must_hold":["The model needs high-quality initial conditions from a data assimilation system.","The weather to be forecast must resemble the reanalysis period used for training."],"echoes":[{"date":"2025-10-20","setting":"CIRA/Colorado State University operations-based verification of GraphCast, Pangu-Weather and other AI models for 2023 tropical cyclones (AIBD-0090)","outcome":"recurred","source":"https://doi.org/10.1175/AIES-D-24-0085.1"}]}}},"provenance":{"kind":"legacy_record","basis":"Editorial assignment.","record_fields":["account","replication"],"source_refs":[]},"review_state":"record_mapped","coverage":"partial"}},"summary":{"total":91,"addressed":91,"known":42,"reviewed":17,"reviewed_complete":16,"reviewed_partial":1,"record_mapped":26,"review_not_assessed":48,"not_assessed":0,"complete":false,"by_state":{"value_present":42,"not_reported":0,"unresolved":49,"not_assessed":0,"not_applicable":0,"withheld":0,"source_unavailable":0}}},"presentation_version":"4707ed45fbfa9d39de8f2463c3682b5c7cb770393de83837a93e4eab3ecf0ee4"}