PipelineJob
Run a pipeline: expand its blocks into units and execute the graph.
The job owns what a job owns -- the database session, the run row, its status transitions -- and the engine owns execution. Between them sits the expansion, which needs the run's id: naming a node's artifacts has to happen before the graph is built, so the row is created here first and the engine never creates anything.
kwargs is a single id, as every job's is: the whole job is serialized
with dill to reach the worker process, which rebuilds its dependencies from
a fresh container, so nothing but plain data can travel in it. The graph is
read from the database inside run.
Methods
get_job_name(self) -> str
PipelineJobGet a descriptive name for the job.
run(self, session_factory: 'sessionmaker' = <function PipelineJob.<lambda> at 0x7f8ea7f72160>) -> None
PipelineJobRun the job.
set_status_as_delivered(self) -> None
PipelineJobNothing to mark: the run row does not exist until the job starts.
set_status_as_error(self) -> None
PipelineJobNothing to mark.
on_cancel(self) -> None
BaseJobCalled in the consumer process after the worker subprocess is killed.
report_progress(self, fraction: Optional[float], message: Optional[str] = None) -> None
BaseJobReport the job's progress to the job queue.
Parameters
- fraction: Optional float
- Completion in the range 0-1, or None when the total work is unknown (the frontend then shows an indeterminate bar).
- message: Optional str
- Short description of the current phase.