Validation¶

A class for validating model predictions against experimental data.

class ionworkspipeline.Validation(objectives: dict[str, FittingObjective | DesignObjective], summary_stats: list[ObjectiveFunction] | None = None)¶

Validate model performance against experimental data.

Evaluates model accuracy by comparing predictions to data using specified objectives and computing summary statistics. Results include time-series comparisons and scalar metrics (RMSE, MAE, etc.).

Parameters¶

objectivesdict[str, FittingObjective | DesignObjective]: Objectives defining what to validate. Keys are objective names, values are objective instances with data and model configuration.
summary_statslist[ObjectiveFunction] | None, default=None: Error metrics to compute. For FittingObjective, defaults to [RMSE, MAE, Max]. For DesignObjective, uses metrics defined by the objective.

Examples¶

Validate voltage predictions:

>>> results = sample_validation.run(sample_parameter_values)
>>> "discharge" in sample_validation.summary_stats
True
>>> sample_validation.summary_stats["discharge"][0]["name"]
'RMSE [mV] (Voltage [V])'

Extends: PipelineElement

See also: Validation — field-level documentation.

export_json(filename='validation_results.json', include_plots: bool = True, plots: list[str] | None = None, plot_kwargs: dict | None = None, backend: str = 'bokeh')¶

Export validation results to JSON format.

Parameters¶

filenamestr

The name of the JSON file to save.

include_plotsbool, default=True

Whether to include plot data in JSON format. If True, plots are stored as JSON objects that can be recreated.

plotslist[str] | None, default=None

The plots to include in the export. If None, model data and error plots will be included. Valid plots are the same as in export_report():

“model data”
“error”
“log error”
“cumulative rmse”
“error moving average”

plot_kwargsdict | None, default=None

Additional keyword arguments for plot creation.

backendstr, default=”bokeh”

Plotting backend to use: “bokeh” (default) or “plotly”. For bokeh, figures can be embedded with Bokeh.embed.embed_item(). For plotly, figures can be loaded with plotly.io.from_json().

Returns¶

None: The JSON file is saved to disk at the specified filename.

Notes¶

The JSON structure is:

{
    "validation_results": {
        "objective_name": {
            "data": {...},  # validation results data
            "plots": [...]  # list of figure JSON
        }
    },
    "summary_stats": {
        "objective_name": [
            {"name": "...", "metric": ...}
        ]
    }
}

export_report(filename='validation_report', include_bokehjs: str | bool = 'cdn', include_plotlyjs: str | bool = 'cdn', backend: str = 'bokeh', plots: list[str] | None = None, plot_kwargs: dict | None = None)¶

Export a report of the validation results.

Parameters¶

filenamestr

The name of the folder where the report will be saved. The main report will be named index.html inside this folder.

include_bokehjsstr | bool, default=’cdn’

Whether to include the bokeh.js library in the report (for bokeh backend). Default is ‘cdn’, which loads the library from the cdn. This reduces file size, but requires an internet connection.

include_plotlyjsstr | bool, default=’cdn’

Whether to include the plotly.js library in the report (for plotly backend). Default is ‘cdn’, which loads the library from the cdn. This reduces file size, but requires an internet connection.

backendstr, default=’bokeh’

Plotting backend to use: “bokeh” (default) or “plotly”.

plotslist[str] | None, default=None

The plots to include in the report. If None, model data and error plots will be included. Valid plots are:

“model data”
Plots model and data voltage vs time.

“error”
Plots error, model voltage - data voltage in mV vs time.

“log error”
Plots log error, log10(abs(model voltage - data voltage)) vs time.

“cumulative rmse”
Plots cumulative RMSE vs time.

“error moving average”
Plots the moving average of the error vs time, with a window of 5 points.

plot_kwargsdict | None, default=None

Additional keyword arguments for plot creation:

window_lengthint, default=5
The window length for the moving average in the error moving average plot.

property failed_objectives: dict[str, str]¶: Formatted error string per objective that failed (read-only).

property finished: bool¶: Whether run() has stored results on this instance.

classmethod from_schema(schema)¶

Construct a Validation from a validated ionworks_schema.Validation.

objectives and summary_stats are forwarded with each nested schema instance unwrapped via its own from_schema (when present), and pre-built pipeline objects passed through unchanged.

Parameters¶

schemaionworks_schema.Validation: Validated schema describing the validation objectives and summary statistics.

Returns¶

Validation: Pipeline-side Validation ready to run.

property results: ValidationResult¶

Results from the last run() (read-only).

Raises¶

UserConfigurationError: If run() has not been called yet. Guard with finished when results may legitimately be absent.

run(parameter_values: ParameterValues, inputs: dict | None = None, executor: Any | None = None) → ValidationResult¶

Execute validation and compute error metrics.

Parameters¶

parameter_valuesiwp.ParameterValues: Model parameters to use for validation simulations.
inputsdict | None, default=None: Input values passed to each objective’s run() method. When omitted, the full objective parameter set is used for backwards compatibility.
executorTaskExecutor | None, default=None: When provided, the per-objective simulations are dispatched across the executor’s worker pool (one task per objective) instead of being run serially in this process. Objectives are independent, so this is an embarrassingly parallel speed-up; results are identical either way. The cheap metric/plot preparation always runs on the driver.

Returns¶

iwp.ValidationResult: Validation results.

Examples¶

>>> sample_validation.run(sample_parameter_values)
{}
>>> "discharge" in sample_validation.summary_stats
True

Notes¶

The returned result is also stored on the instance and exposed through the read-only properties below (each raises until run() has been called; check finished first when that is possible):

validation_resultsdict[str, dict[str, float]]

Time-series comparisons by objective name.

For FittingObjective:

Contains model vs. data with keys like “Model voltage [V]”, “Processed data voltage [V]”, “Error [mV]”, “Time [s]”, etc.

For DesignObjective:

Contains model time-series outputs like “Time [s]”, “Current [A]”, and simulated variables.

summary_statsdict[str, list[dict[str, float]]]

Scalar error metrics by objective name.

For FittingObjective:

List of dicts with ‘name’ (e.g., “RMSE [mV]”) and ‘metric’ (value).

For DesignObjective:

Single-item list with dict mapping metric names to values.

property summary_stats: dict¶: Scalar error metrics by objective name (read-only).

to_config() → dict¶

Convert the Validation back to parser configuration format.

Returns¶

dict: Configuration dictionary that can be passed to parse_validation

property validation_results: dict¶: Time-series comparisons by objective name (read-only).