Task Runners

Task graphs defined by DV Flow can be run in many ways. Because the topology of task graphs is known before execution, task graphs can be evaluated both statically and dynamically.

DV Flow Manager defines the RunnerBackend interface to enable support for multiple mechanisms to dynamically execute task graphs.

Runner Backends

DFM ships with a local runner that schedules tasks across the available cores on a single machine. Additional backends can dispatch tasks to remote workers on LSF or SLURM clusters.

See the Task Runners user guide for details on selecting and configuring runners, and Runner Config for the full configuration reference.

class dv_flow.mgr.runner_backend.RunnerBackend

Abstract execution backend.

Implementations provide different strategies for executing tasks: local (in-process via jobserver), LSF, SLURM, etc.

abstractmethod async acquire_slot() → None

Acquire an execution slot (analogous to jobserver token).

Blocks if the backend is at capacity.

async cancel_inflight() → None

Cancel all tasks currently in flight.

Called during cleanup (e.g. SIGINT) to inform the backend that any outstanding tasks should be aborted. The default is a no-op; remote backends override this to notify the daemon.

abstractmethod async execute_task(request: TaskExecRequest) → TaskDataResult

Execute a single task. Blocks until the task completes.

The backend is responsible for: - Selecting or launching an appropriate worker - Transmitting the request - Waiting for and returning the result - Handling worker failures (retry on a different worker)

property is_remote: bool

True if this backend dispatches tasks to remote workers.

abstractmethod async release_slot() → None

Release an execution slot.

abstractmethod async start() → None

Initialize backend (connect to daemon, start pool, etc.).

abstractmethod async stop() → None

Shutdown backend (drain workers, release resources).

class dv_flow.mgr.runner_backend_local.LocalBackend(jobserver: JobServer | None = None)

Execute tasks locally using the existing jobserver.

For LocalBackend, TaskNodeLeaf._run_task continues to call the callable directly (no serialization overhead). The backend is only consulted for slot acquisition.

async acquire_slot() → None

Acquire a jobserver token.

async execute_task(request: TaskExecRequest) → TaskDataResult

LocalBackend uses in-process execution; this should not be called.

The local path goes through TaskNodeLeaf._run_task directly.

property is_remote: bool

True if this backend dispatches tasks to remote workers.

async release_slot() → None

Release a jobserver token.

async start() → None

No-op for local execution.

async stop() → None

No-op for local execution.