Skip to main content

types

Batch execution types and data structures for protocols.

This module contains the core data structures and constants used for batch execution in federated protocols, extracted from the main base.py to improve modularity.

Classes​

BatchConfig​

class BatchConfig(    batch_size: int,    data_splitter: DatasetSplitter,    datasource: FileSystemIterableSource,    original_file_names_override: Optional[list[str]],    is_final_batch: bool = False,):

Holds batch configuration and state.

Arguments

  • batch_size: The size of each batch.
  • data_splitter: The dataset splitter to use for splitting the data.
  • datasource: The datasource from which to read the data.
  • original_file_names_override: Optional. Override for the original file names used in the datasource. Used for restoring selected_file_names_override on the datasource the at the end of the task.
  • is_final_batch: Whether this is the final batch. Defaults to False.

Attributes

  • batch_size: The size of each batch.
  • data_splitter: The dataset splitter to use for splitting the data.
  • datasource: The datasource from which to read the data.
  • original_file_names_override: Optional. Override for the original file names used in the datasource. Used for restoring selected_file_names_override on the datasource at the end of the task.
  • current_batch: The current batch number.
  • current_files_iterator: An iterator over the filenames for the current batch.
  • is_final_batch: Whether this is the final batch.
  • current_batch_files: The list of files in the current batch.
  • has_new_files: Whether any new files have been found (for run_on_new_data_only).
  • total_files_checked: Total number of files checked so far.
  • _sent_batch_count_update: Whether we've sent batch count update to modeller.
  • failed_batches: Dictionary mapping batch numbers to their failure exceptions.
  • consecutive_failures: Count of consecutive batch failures.
  • successful_batches: List of batch numbers that processed successfully.
  • file_level_errors: Dictionary mapping file paths to their failure exceptions.
  • individual_file_results: Dictionary mapping file paths to individual test results.

ProtocolState​

class ProtocolState(    execute_final_step: bool = False,    termination_reason: Optional[TerminationReason] = None,    reduce_step_kwargs: Optional[_StrAnyDict] = None,):

Context for managing batch execution state and early termination.

Arguments

  • execute_final_step: Whether to execute the final step of the protocol. Defaults to False.
  • termination_reason: Optional. The reason for terminating the protocol execution early. Defaults to None.
  • reduce_step_kwargs: Optional. Keyword arguments for the final reduce step. Defaults to None.

Variables​

  • static execute_final_step : bool
  • static reduce_step_kwargs : Optional[dict]

TerminationReason​

class TerminationReason(    value, names=None, *, module=None, qualname=None, type=None, start=1,):

Enumeration of protocol termination reasons.

Ancestors​

Variables​

  • static FINAL_BATCH
  • static LIMITS_EXCEEDED
  • static SINGLE_BATCH