Europe's forecast center tests Pangu-Weather beside its own model

ECMWF ran Huawei's Pangu-Weather from its own analyses and found accuracy comparable to its IFS model, with better summer temperature-event forecasts in Europe.

Benefit status
Intended Read outcome assessment
Evidence
Claimed by the parties
Source published
Sources
1 publication
Download data CiteSuggest a correction

Benefits

Benefit

Pangu-Weather discriminated events better than the IFS

Awaiting review

Outcome identifiers

Unresolved

Not reviewedUnresolved

Category

Improved Task Accuracy

Awaiting review

Status

Intended. ECMWF scored Pangu-Weather forecasts for past periods against its own model and against observations, which is a retrodiction, and the authors did not measure any use of the forecasts in operations or warnings. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Beneficiary groups

Organizations.

Awaiting review

Beneficiaries

Unresolved

Not reviewedUnresolved

Reach

Unresolved

Not reviewedUnresolved

Distribution and access

Unresolved

Not reviewedUnresolved

Duration

Unresolved

Not reviewedUnresolved

Expected?

Yes

Awaiting review

Costs and harms

Unresolved

Not reviewedUnresolved

AI system

System

Pangu-Weather (PanguWeather, PGW)

Awaiting review

Model

This is an explicit unresolved assignment: the sources on record do not settle this question.

Not reviewedUnresolved

Release

Unresolved

Not reviewedUnresolved

Provider

Unresolved

Not reviewedUnresolved

Role

Unresolved

Not reviewedUnresolved

Configuration

Unresolved

Not reviewedUnresolved

Inputs and outputs

Unresolved

Not reviewedUnresolved

Autonomy

Unresolved

Not reviewedUnresolved

Workflow

ECMWF ran Pangu-Weather from its own operational IFS analyses. Both models were verified with the same scores against analyses and station observations.

Awaiting review

Human oversight

Unresolved

Not reviewedUnresolved

Relation to AI

ECMWF started Pangu-Weather and the IFS from the same operational analyses and verified both with the same tools, so the authors isolate the skill of the machine-learning model.

Awaiting review

AI contribution

ECMWF started Pangu-Weather and the IFS from the same operational analyses and verified both with the same tools, so the authors isolate the skill of the machine-learning model.

Awaiting review

Measurement

Studies

Controlled observational. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Study design

Controlled observational. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Measurement

Standard verification scores (correlation, RMSE, area under the ROC curve, cyclone position and pressure error) against IFS analyses, synoptic observations and IBTrACS. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Setting and sample

Ten-day forecasts from 1 March 2022 to 28 February 2023, with 2018 for tropical cyclones; Northern Hemisphere and Europe. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Measures

Upper-air and 2-meter temperature skill; event discrimination; cyclone track and intensity error. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Comparison

Operational IFS forecasts from the same initial conditions. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Results

Comparable global skill; better summer event discrimination for the 75th to 90th percentiles; similar cyclone track error; worse cyclone intensity. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Additional value from AI

Comparable accuracy at much lower computing cost; cost not quantified in the paper. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Limitations

Deterministic forecasts from one model; smooth fields and growing bias; poor cyclone intensity; no measure of use by forecasters. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Denominators

Research runs only. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Materials for replication

Model weights public for noncommercial use; ECMWF toolkit public; ECMWF forecasts available. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Money

Kind of value

The kind of monetary value is unclear.

Not reviewedUnresolved

Amount

The monetary value is not established.

Not reviewedUnresolved

Currency and prices

The currency and price basis are unclear.

Not reviewedUnresolved

Period and unit

The period and unit of the monetary estimate are unclear.

Not reviewedUnresolved

Realization

Whether the monetary value was realized is unclear.

Not reviewedUnresolved

Recipients and payers

The recipients and payers are unclear.

Not reviewedUnresolved

Gross or net

Whether the monetary figure includes costs is unclear.

Not reviewedUnresolved

Included costs

The costs included in the estimate are unclear.

Not reviewedUnresolved

Comparison

The financial comparison is not established.

Not reviewedUnresolved

Method

The method used to calculate monetary value is unclear.

Not reviewedUnresolved

Return and payback

The return on investment or payback period is not established.

Not reviewedUnresolved

People and places

Participants

Developer: Huawei Cloud. Operator: European Centre for Medium-Range Weather Forecasts (ECMWF), in an operational-like test.

Awaiting review

Jurisdictions

Benefit: ECMWF, Reading, United Kingdom, and Bonn, Germany; verification over the Northern Hemisphere and Europe. Developer: China. Operator: United Kingdom.

Awaiting review

Ownership and control

Unresolved

Not reviewedUnresolved

Dependencies

Unresolved

Not reviewedUnresolved

Cross-border connections

Unresolved

Not reviewedUnresolved

Geopolitical implications

The geopolitical implications are unclear.

Not reviewedUnresolved

Follow-up

Conditions for replication

The model needs high-quality initial conditions from a data assimilation system. The weather to be forecast must resemble the reanalysis period used for training.

Awaiting review

Replication attempts

1 follow-up entry. Benefit recurred: 1.

Awaiting review

Later findings

Unresolved

Not reviewedUnresolved

Availability

Unresolved

Not reviewedUnresolved

Institutional decision

A relevant institutional decision has not been established.

Not reviewedUnresolved

Grounds

The grounds for an institutional decision are unclear.

Not reviewedUnresolved

Action and result

The result of an institutional decision is not established.

Not reviewedUnresolved

Success criterion

The criterion for a successful institutional decision is unclear.

Not reviewedUnresolved

Evidence

Source numbers

AIBD-0657-source-0001

Awaiting review

Citations

Numbered references below.

Awaiting review

Language

Unresolved

Not reviewedUnresolved

Source relationships

Evaluator

Awaiting review

Original evidence

Unresolved

Not reviewedUnresolved

Access and versions

Unresolved

Not reviewedUnresolved

Supported findings

Unresolved

Not reviewedUnresolved

Evidence strength

The evaluator's account. Sources: the evaluators' peer-reviewed study. The cited evidence includes no independent source and no finding by a regulator or court. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Missing evidence

Unresolved

Not reviewedUnresolved

Grades

Assessed benefits

Organizations.

Awaiting review

Depth

The revised record no longer states this information.

Not reviewedUnresolved

The record revision removed the mapped value.

Reasons

Organizations: The study compares forecast skill on past periods and reports no realized gain for ECMWF or for any user of the forecasts.

Awaiting review

Observed or intended

Intended. ECMWF scored Pangu-Weather forecasts for past periods against its own model and against observations, which is a retrodiction, and the authors did not measure any use of the forecasts in operations or warnings. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Rubric

Unresolved

Not reviewedUnresolved

Assessor and date

Unresolved

Not reviewedUnresolved

Alternative assessments

Unresolved

Not reviewedUnresolved

Fill

The evaluator's account

Awaiting review

Entry details

Kind

The case describes an evaluation or a study. A verification study by a forecast center independent of the developer. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Field

Weather forecasting.

Awaiting review

Purpose

Pangu-Weather was built to produce global medium-range weather forecasts at a small fraction of the computing cost of physics-based models. Forecast accuracy in an operational setting is that purpose.

Awaiting review

Dates

Source published: 6 June 2024. Updated: 29 October 2026.

Awaiting review

Stage

The system was tested in a laboratory. ECMWF ran the model in an operational-like setting for research; it did not issue forecasts from it. [1]

Fully reviewed

Review date: 29 October 2026.

Sources: [1]

Version

SHA-256: 31243383b1b5…

Awaiting review

SHA-256 of the public case, its assessments, and its displayed answers and notes, excluding its own identifier and the derived coverage summary.

References

Other identifiers

Unresolved

Not reviewedUnresolved

Category mappings

Unresolved

Not reviewedUnresolved

Citable summary

Unresolved

Not reviewedUnresolved

Revisions

2 revisions. Latest: 29 October 2026.

Awaiting review

Sources

  1. Peer-reviewed study Evaluator

    Ben Bouallègue Z, Clare MCA, Magnusson L, et al. The rise of data-driven weather forecasting: a first statistical assessment of machine learning-based weather forecasts in an operational-like context. Bull Am Meteorol Soc. 2024;105(6):E864-E883.

Collections: Cross-border cases.

Cite this case

BibTeX Download BibTeX

Version 31243383b1b5, archived in the edition of 29 October 2026

Benefit marks

Depth

The number and color show the depth of a benefit.

  1. A convenience, or time or money saved.
  2. A measurable improvement in a result or ability.
  3. A lasting change, an ability restored, or a risk removed.
  4. A serious harm averted or a serious condition resolved in time.
  5. A life or livelihood preserved, or a change documented across a population.

Beneficiary

  • Circle: a person
  • Square: an organization
  • Triangle: society

Evidence

  • Solid. A court or regulator found a benefit, or researchers independently studied it.

  • Hatched. Independent reporting.

  • Outline. Claimed by the parties. Only the people or organizations involved, or sources whose independence has not been assessed, support the benefit.

The fill changes with the evidence. The depth color stays the same.

How benefits are assessed