Skip to main content

schema_management

Schema management utilities for the Pod.

Classes​

PrefectProcessManager​

class PrefectProcessManager():

Manages Prefect flows in a separate process.

Methods​

start_flow​

def start_flow(    self,    base_datasources: dict[str, DatasourceContainer],    hub_config: HubConfig,    pod_public_key: RSAPublicKey,    _access_manager_public_key: RSAPublicKey,    pod_name: str,) ‑> None:

Start Prefect flow in a separate process.

stop​

def stop(self) ‑> None:

Stop the Prefect process.

SchemaGenerationHooks​

class SchemaGenerationHooks():

Prefect hooks for schema generation.

Static methods​

on_schema_worker_completion​

def on_schema_worker_completion(tsk: Task, run: TaskRun, state: State) ‑> None:

Log completion of schema worker task.

on_schema_worker_failure​

def on_schema_worker_failure(tsk: Task, run: TaskRun, state: State) ‑> None:

Log failure of schema worker task.

SchemaManagement​

class SchemaManagement():

Schema management utilities for the Pod.

Static methods​

is_prefect_server_healthy​

def is_prefect_server_healthy() ‑> bool:

Check if the Prefect server is up and running.

schema_worker​

async def schema_worker(    datasource_config: MinimalDatasourceConfig,    schema_generation_config: MinimalSchemaGenerationConfig,    schema_upload_config: MinimalSchemaUploadConfig,    hub_config: HubConfig,) ‑> tuple[str, str]:

Process each record in the dataset with the ability to cancel.

Arguments

  • datasource_config: The datasource config to generate schema for.
  • schema_generation_config: The schema generation config.
  • schema_upload_config: The schema upload config.
  • hub_config: The hub config.

Returns The datasource name and the schema as a JSON string.

start_prefect_flow​

def start_prefect_flow(    base_datasources: dict[str, DatasourceContainer],    hub_config: HubConfig,    pod_public_key: RSAPublicKey,    _access_manager_public_key: RSAPublicKey,    pod_name: str,) ‑> None:

Start Prefect flow in a separate process.

stop_prefect_flow​

def stop_prefect_flow() ‑> None:

Stop Prefect flow process.