Data Loading

A class for loading data for use in other Ionworks packages.

class ionworks.data.load.DataLoader(time_series: DataFrame | DataFrame | dict, steps: DataFrame | DataFrame | dict | None = None, **kwargs)

Bases: object

Load time-series data with steps, or plain tabular data such as OCP curves.

data and steps are Polars DataFrames; inputs may be pandas or Polars.

Parameters:

• time_series (pd.DataFrame | pl.DataFrame | dict) – The data to load. Can be a Pandas/Polars DataFrame or a dict.

• steps (pd.DataFrame | pl.DataFrame | dict | None, optional) – Step information. When None, the loader operates in simple (no-steps) mode.

• **kwargs –

  Options passed directly or via an options dict. The set below is closed — an unrecognised key raises rather than being ignored, so a misspelt option cannot silently skip the preprocessing you asked for. Supported keys:

  When steps are provided:

  • first_step, last_stepint | str | dict

    A step index, a SQL query selecting one row of the steps table, or the deprecated cycle/step dict.

  • first_step_dict, last_step_dict : deprecated spellings of the above

  Always:

  • capacity_columnstr

    Name of the column in time_series to use as the capacity axis (copied to "Capacity [A.h]").

  • transformsdict with any of:

    • rest_to_ocpbool

      See transform_rest_to_ocp() for details.

    • sortbool

      See sort_capacity_and_ocp() for details.

    • remove_duplicatesbool

      See remove_duplicate_ocp() for details.

    • remove_extremesbool

      See remove_ocp_extremes() for details.

    • filtersdict

      See filter_data() for details.

    • interpolatefloat | list[float]

      See interpolate_data() for details. An ndarray is accepted and normalised to a list.

    • keep_first_ocp_pointbool

      If True, prepend the first point (see transform_rest_to_ocp()). Default False. Ignored if rest_to_ocp is False.

__init__(time_series: DataFrame | DataFrame | dict, steps: DataFrame | DataFrame | dict | None = None, **kwargs)

__getitem__(key: str) → Series

generate_experiment(use_cv: bool = False, termination: str = 'duration', period: str | float | None = None, split_cc_cv: bool = True) → pybamm.Experiment

Generate a PyBaMM experiment from the loaded step information.

Parameters:

• use_cv (bool, optional) – Emit constant-voltage steps as voltage holds rather than as a current interpolant. Default False.

• termination (str, optional) –

  What ends each step. Default "duration".

  • "duration": every step ends after its measured Duration [s].

  • "events": every step ends where the measured step ended, in the variable it was controlling — a constant-current or constant-power step at its final voltage, a voltage hold at its final current — and carries no duration, so its length becomes the model’s own. A capacity disagreement between model and cell then reads as capacity to that endpoint rather than as a residual voltage. Rests and steps emitted as a current interpolant keep their duration either way.

  "events" replays endpoints, not the cycler’s intent: a step the cycler ended on a time limit is still replayed as ending at the voltage it reached, because the steps table cannot tell the two apart. Use it for endpoint-driven protocols (rate capability, CC-CV cycling) and "duration" for time-driven ones. Where a step’s voltage barely moves, such as a short pulse, the endpoint can already be satisfied at the model’s initial state; PyBaMM warns and skips such a step rather than solving it.

• period (str or float, optional) – Sampling period for the emitted steps, as pybamm.Experiment takes it. "from data" gives each step the measurement’s own mean sample interval. Default None, which leaves the output grid to PyBaMM.

• split_cc_cv (bool, optional) – Emit a step no type fits that ran constant current and then held its voltage, as cyclers often log a CC-CV charge, as a constant-current step followed by a constant-voltage step, rather than as a current interpolant. Default True. A split step becomes two experiment steps, so every later step of its cycle sits one position further along than its row in the steps table; code that reads the experiment’s steps by steps-table position has to allow for it.

Returns:

One PyBaMM cycle per run of equal "Cycle count" in the steps table. A table without that column has no cycle structure to recover and becomes a single cycle. The "Cycle count" behind each cycle is left in experiment_cycle_counts, since the experiment itself numbers its cycles by position, and the steps-table row behind each step in experiment_step_sources.

Return type:

pybamm.Experiment

plot_data(show: bool = False) → tuple[plt.Figure, plt.Axes]

Plot voltage vs time data from the loaded experiment.

static filter_data(data: DataFrame, filters: dict) → DataFrame

Filter a Polars DataFrame using the specified filter functions.

Each key in filters is a column name; the value must include filter_type (e.g. "savgol") and any parameters. Used by the filters transform option.

static interpolate_data(data: DataFrame, knots: float | ndarray, x_column: str = 'Time [s]') → DataFrame

Interpolate a Polars DataFrame using np.interp.

If knots is a float, data is resampled at that regular interval along x_column. If an array, interpolated at those knots. Used by the interpolate transform option.

Only numeric columns are interpolated. Non-numeric (e.g. Utf8/String) columns are skipped and do not appear in the returned DataFrame.

classmethod from_local(data_path, options=None, use_polars=True)

Load data from local filesystem.

Parameters:

• data_path (str) – Path to the directory containing time_series.parquet (or time_series.csv) and optionally steps.parquet (or steps.csv).

• options (dict | None, optional) – Options to pass to the DataLoader constructor.

• use_polars (bool, optional) – If True (default), read with Polars. If False, read with Pandas (data is still stored as Polars internally).

Return type:

DataLoader

experiment_cycle_counts: list | None = None

The "Cycle count" behind each cycle of the last experiment generate_experiment() produced, which numbers its cycles by position. None before one is generated, and without that column.

experiment_step_sources: list[list[int]] | None = None

Per cycle, the "Start index" of the row behind each step, since a CC-CV split or a skipped zero-duration step shifts positions.

__init__(time_series: DataFrame | DataFrame | dict, steps: DataFrame | DataFrame | dict | None = None, **kwargs)

property data: DataFrame

Time-series data as a Polars DataFrame. Use .data.to_pandas() for pandas.

__getitem__(key: str) → Series

static filter_data(data: DataFrame, filters: dict) → DataFrame

Filter a Polars DataFrame using the specified filter functions.

Each key in filters is a column name; the value must include filter_type (e.g. "savgol") and any parameters. Used by the filters transform option.

static interpolate_data(data: DataFrame, knots: float | ndarray, x_column: str = 'Time [s]') → DataFrame

Interpolate a Polars DataFrame using np.interp.

If knots is a float, data is resampled at that regular interval along x_column. If an array, interpolated at those knots. Used by the interpolate transform option.

Only numeric columns are interpolated. Non-numeric (e.g. Utf8/String) columns are skipped and do not appear in the returned DataFrame.

property steps: DataFrame | None

Step summary as a Polars DataFrame, or None. Use .steps.to_pandas() for pandas.

property initial_voltage

Initial voltage (from first step or previous step end). Lazy-loads steps when from_db.

set_processed_internal_state(*, transforms=None, measurement_id=None, capacity_column=None, first_step=None, last_step=None, original_time_series=None, original_steps=None)

Set internal state when constructing from processed data. Used by from_processed_data.

original_time_series and original_steps may be pandas or Polars DataFrames (or None); they are stored as Polars internally for config export.

transform_rest_to_ocp()

Extract OCP from rest steps: take the last data point of each rest.

Filters steps to those with Step type == "Rest", computes cumulative net capacity (discharge/charge reset per step), then builds one OCP point (capacity, voltage) at the end of each rest. If transforms["keep_first_ocp_point"] is True, prepends the first row of the time series as an extra OCP point. Replaces data with the OCP table and clears steps.

static remove_duplicate_ocp(data: DataFrame, capacity_column_name='Capacity [A.h]') → DataFrame

Remove duplicate capacity and voltage points.

Keeps first occurrence of each unique capacity and each unique voltage. Used by the remove_duplicates transform option.

static sort_capacity_and_ocp(data: DataFrame) → DataFrame

Sort OCP data so voltage is decreasing and capacity is increasing.

Ensures a single capacity column (normalized to start at 0 and non-decreasing), removes duplicates, and reverses rows if voltage is increasing. Used by the sort transform option.

static remove_ocp_extremes(data: DataFrame) → DataFrame

Remove OCP points at extremes where d²V/dQ² is zero.

Trims the capacity–voltage curve to the range where the second derivative of voltage with respect to capacity is non-zero. Used by the remove_extremes transform option.

calculate_dUdQ_cutoff(method: str = 'explicit', show_plot: bool = False, options: dict | None = None) → float

Calculate the cut-off for dUdQ based on the data.

Parameters:

• method (str, optional) – Method to use for calculating the cut-off. Options are: - “explicit” (default): Uses explicit method based on data range - “quantile”: Uses quantile-based method - “peaks”: Uses peak detection method

• show_plot (bool, optional) – Whether to show a plot of the dUdQ values with the cut-off.

• options (dict, optional) – Dictionary of options to pass to the method.

Returns:

Cut-off for dUdQ

Return type:

float

calculate_dQdU_cutoff(method: str = 'explicit', show_plot: bool = False, options: dict | None = None) → float

Calculate the cut-off for dQdU based on the data.

Parameters:

• method (str, optional) – Method to use for calculating the cut-off. Options are: - “explicit” (default): Uses explicit method based on data range - “quantile”: Uses quantile-based method - “peaks”: Uses peak detection method

• show_plot (bool, optional) – Whether to show a plot of the dQdU values with the cut-off.

• options (dict, optional) – Dictionary of options to pass to the method.

Returns:

Cut-off for dQdU

Return type:

float

to_local() → DataLoader

Convert a DB-backed loader into a fully local one.

Ensures all lazy data (steps and time series) is fetched, then removes the measurement ID so that to_config() serialises the data inline instead of as a db: reference.

Returns self for chaining, e.g. loader.to_local().to_config().

to_config(filter_data: bool = True) → dict

Convert the DataLoader back to parser configuration format.

Parameters:

filter_data (bool, optional) – If True (default) and steps are present, saves the filtered data rather than the original unfiltered data.

Returns:

Configuration dictionary that can recreate this DataLoader.

Return type:

dict

property start_idx: int

property end_idx: int

generate_experiment(use_cv: bool = False, termination: str = 'duration', period: str | float | None = None, split_cc_cv: bool = True) → pybamm.Experiment

Generate a PyBaMM experiment from the loaded step information.

Parameters:

• use_cv (bool, optional) – Emit constant-voltage steps as voltage holds rather than as a current interpolant. Default False.

• termination (str, optional) –

  What ends each step. Default "duration".

  • "duration": every step ends after its measured Duration [s].

  • "events": every step ends where the measured step ended, in the variable it was controlling — a constant-current or constant-power step at its final voltage, a voltage hold at its final current — and carries no duration, so its length becomes the model’s own. A capacity disagreement between model and cell then reads as capacity to that endpoint rather than as a residual voltage. Rests and steps emitted as a current interpolant keep their duration either way.

  "events" replays endpoints, not the cycler’s intent: a step the cycler ended on a time limit is still replayed as ending at the voltage it reached, because the steps table cannot tell the two apart. Use it for endpoint-driven protocols (rate capability, CC-CV cycling) and "duration" for time-driven ones. Where a step’s voltage barely moves, such as a short pulse, the endpoint can already be satisfied at the model’s initial state; PyBaMM warns and skips such a step rather than solving it.

• period (str or float, optional) – Sampling period for the emitted steps, as pybamm.Experiment takes it. "from data" gives each step the measurement’s own mean sample interval. Default None, which leaves the output grid to PyBaMM.

• split_cc_cv (bool, optional) – Emit a step no type fits that ran constant current and then held its voltage, as cyclers often log a CC-CV charge, as a constant-current step followed by a constant-voltage step, rather than as a current interpolant. Default True. A split step becomes two experiment steps, so every later step of its cycle sits one position further along than its row in the steps table; code that reads the experiment’s steps by steps-table position has to allow for it.

Returns:

One PyBaMM cycle per run of equal "Cycle count" in the steps table. A table without that column has no cycle structure to recover and becomes a single cycle. The "Cycle count" behind each cycle is left in experiment_cycle_counts, since the experiment itself numbers its cycles by position, and the steps-table row behind each step in experiment_step_sources.

Return type:

pybamm.Experiment

generate_interpolant() → pybamm.Interpolant

Generate a PyBaMM interpolant from the loaded step information.

plot_data(show: bool = False) → tuple[plt.Figure, plt.Axes]

Plot voltage vs time data from the loaded experiment.

classmethod from_local(data_path, options=None, use_polars=True)

Load data from local filesystem.

Parameters:

• data_path (str) – Path to the directory containing time_series.parquet (or time_series.csv) and optionally steps.parquet (or steps.csv).

• options (dict | None, optional) – Options to pass to the DataLoader constructor.

• use_polars (bool, optional) – If True (default), read with Polars. If False, read with Pandas (data is still stored as Polars internally).

Return type:

DataLoader

classmethod from_db(measurement_id: str, options: dict | None = None, use_cache: bool = True, client=None, time_range: dict | None = None) → DataLoader

Load data from the Ionworks database (lazy loading).

Data is fetched on demand: accessing .initial_voltage or .steps loads only the steps table (small payload) via client.cell_measurement.steps(measurement_id). Accessing .data loads the time series as well via client.cell_measurement.time_series(measurement_id), after steps if needed for slicing. This allows reading e.g. initial voltage without downloading the full time series.

Parameters:

• measurement_id (str) – The ID of the measurement to load from the database.

• options (dict | None, optional) – Options to pass to the DataLoader constructor.

• use_cache (bool, optional) – If True (default), use local file cache to avoid repeated API calls. Set to False to force a fresh load from the database.

• client (ionworks.api.Client | None, optional) – Pre-configured Ionworks client. If not provided, a default Client() is created (using env vars).

• time_range (dict | None, optional) – Optional {"start": seconds, "end": seconds} window of elapsed Time [s] values, applied after first_step / last_step have chosen their steps and written into to_config(). It pins a re-run to the data the first run saw, so later appends do not change the answer. A property of the pipeline config, not of the measurement’s own start/end times.

Return type:

DataLoader

classmethod from_processed_data(data, steps, initial_voltage, start_idx, end_idx)

Create a DataLoader from already-processed data, bypassing __init__.

Parameters:

• data (pd.DataFrame | pl.DataFrame) – The processed time series data.

• steps (pd.DataFrame | pl.DataFrame | None) – The processed steps data (or None).

• initial_voltage (float) – The initial voltage value.

• start_idx (int) – The start index for the data.

• end_idx (int) – The end index for the data.

Return type:

DataLoader

copy() → DataLoader

Create a copy of the DataLoader instance.

slice_to_steps(first_step_idx: int, last_step_idx: int) → DataLoader

Create a new DataLoader containing only the given step range.

Only requires the steps table to be loaded — the time-series data remains lazy (not loaded) when the source DataLoader is lazy.

Parameters:

• first_step_idx (int) – Row index of the first step in the steps table (0-based).

• last_step_idx (int) – Row index of the last step in the steps table (0-based, inclusive).

Return type:

DataLoader