Runner Backend API

The RunnerBackend interface enables support for multiple mechanisms to dynamically execute task graphs. DFM ships with a local backend that schedules tasks across the available cores on a single machine; additional backends can dispatch tasks to remote workers on LSF or SLURM clusters.

See Task Runners for the user-facing guide on selecting and configuring runners, and Runner Config for the full configuration field reference.

class dv_flow.mgr.runner_backend.RunnerBackend

Abstract execution backend.

Implementations provide different strategies for executing tasks: local (in-process via jobserver), LSF, SLURM, etc.

abstractmethod async acquire_slot() → None

Acquire an execution slot (analogous to jobserver token).

Blocks if the backend is at capacity.

async cancel_inflight() → None

Cancel all tasks currently in flight.

Called during cleanup (e.g. SIGINT) to inform the backend that any outstanding tasks should be aborted. The default is a no-op; remote backends override this to notify the daemon.

abstractmethod async execute_task(request: TaskExecRequest) → TaskDataResult

Execute a single task. Blocks until the task completes.

The backend is responsible for: - Selecting or launching an appropriate worker - Transmitting the request - Waiting for and returning the result - Handling worker failures (retry on a different worker)

property is_remote: bool

True if this backend dispatches tasks to remote workers.

abstractmethod async release_slot() → None

Release an execution slot.

abstractmethod async start() → None

Initialize backend (connect to daemon, start pool, etc.).

abstractmethod async stop() → None

Shutdown backend (drain workers, release resources).

class dv_flow.mgr.runner_backend_local.LocalBackend(jobserver: JobServer | None = None)

Execute tasks locally using the existing jobserver.

For LocalBackend, TaskNodeLeaf._run_task continues to call the callable directly (no serialization overhead). The backend is only consulted for slot acquisition.

async acquire_slot() → None

Acquire a jobserver token.

async execute_task(request: TaskExecRequest) → TaskDataResult

LocalBackend uses in-process execution; this should not be called.

The local path goes through TaskNodeLeaf._run_task directly.

property is_remote: bool

True if this backend dispatches tasks to remote workers.

async release_slot() → None

Release a jobserver token.

async start() → None

No-op for local execution.

async stop() → None

No-op for local execution.