Run an evaluation job¶

This guide walks through running an evaluation on a trained checkpoint: launching the job, enabling visualisations, and finding the outputs.

Prerequisites¶

Make sure IceNet-MP is installed and that you have a trained checkpoint — either from a training run or downloaded from shared storage.

1. Get a checkpoint¶

From a training run¶

Checkpoints are saved to ${BASE_DIR}/training/wandb/run-<date>-<id>/checkpoints/<name>.ckpt after training. Pick the checkpoint you want to evaluate.

From shared storage (HPC)¶

Pre-trained checkpoints are available on Baskerville, DAWN, and Isambard-AI. Ask a team member for the path.

2. Create a local config¶

If you do not already have a local config from a training run, create one at icenet_mp/config/<your-name>.local.yaml. See Train a model — Create a local config for details.

3. Run evaluate¶

See the evaluate command reference for full option details, then run:

uv run imp evaluate --config-name <your-name>.local --checkpoint PATH_TO_CHECKPOINT

Enabling visualisations¶

By default, all visualisations are enabled (see icenet_mp/config/evaluate/callbacks/plotting.yaml). To disable forecast plots, set make_static_plots and make_video_plots to false in your local config:

evaluate:
  callbacks:
    plotting:
      make_static_plots: false
      make_video_plots: false

Plots of the raw input data are also enabled by default. To disable them, set:

evaluate:
  callbacks:
    plotting:
      make_input_plots: false

4. Check results in W&B¶

Once evaluation completes, the run appears in the W&B project evaluate under the turing-seaice entity at wandb.ai.

Key	Contents
`output_static`	Static images of forecast output.
`output_video`	Animated forecast output.
`input_static`	Static images of the raw input data (if `make_input_plots: true`).
`input_video`	Animated raw input data (if `make_input_plots: true`).
`Custom Charts`	Per-forecast-day metrics, allowing skill to be assessed at longer lead times.

How to interpret forecast days and example videos¶

The per-forecast-day charts and the example videos summarise different dimensions of the same evaluation run.

For each valid start date in the configured evaluation period, the model produces predictions for each of the next n_forecast_steps timesteps. The value of each metric is averaged across all start dates to give a single value. In other words, the value at, for example, forecast day 3, tells you the average value of that metric at 3 days into the future across all start dates in the evaluation period.

The example videos use a small subset of representative start dates spread across the evaluation period. Each video contains the full forecast sequence for the selected start date. For example, when running with n_forecast_steps=7, each video will have 7 frames, one for each forecast day.

Custom Charts show performance as a function of forecast day, aggregated across the evaluation period.
output_video shows a small number of representative forecast sequences from within that period.

This distinction is useful when checking whether skill degrades with lead time without generating a video for every possible forecast start date.