######
Daemon
######

The DFM daemon is a background process that manages a pool of worker
processes across multiple ``dfm run`` invocations. This amortizes the
startup cost of cluster job submission (LSF, SLURM) and keeps warm
workers available for immediate task dispatch.

Starting the Daemon
===================

::

    dfm daemon start

The daemon writes its state to ``<project>/.dfm/daemon.json`` and
listens for client connections on a Unix socket at
``<project>/.dfm/daemon.sock``.

Options:

``--runner <name>``
    Runner backend to use (e.g. ``local``, ``lsf``).

``--pool-size <n>``
    Maximum number of worker processes.

``--monitor``
    Attach the monitor TUI after starting (see below).

Stopping the Daemon
===================

::

    dfm daemon stop

Sends a shutdown request to the running daemon. All workers are
terminated and the state file is removed.

Checking Status
===============

::

    dfm daemon status
    dfm daemon status --json

Displays the current state: PID, number of workers, pending tasks,
and worker details. Use ``--json`` for machine-readable output.

Auto-Discovery from ``dfm run``
===============================

When ``dfm run --runner lsf`` is invoked, the runner backend
automatically looks for ``<project>/.dfm/daemon.json``:

1. If found and the PID is alive, ``dfm run`` connects to the daemon
   and proxies tasks through it. Workers are shared across all
   connected ``dfm run`` processes.

2. If not found, ``dfm run`` can start an ephemeral pool for the
   duration of the run (workers are terminated when ``dfm run`` exits).

Multi-Client Usage
==================

The daemon accepts connections from multiple ``dfm run`` processes
simultaneously. Tasks from all clients are dispatched to the shared
worker pool. Each task result is routed back to the originating client.
This allows multiple regression runs to share a single warm pool.

Stale State Files
=================

If the daemon crashes without cleanup, a stale ``daemon.json`` may
remain. Both ``dfm daemon status`` and ``dfm run`` detect stale files
(by checking if the PID is alive) and handle them gracefully:

- ``dfm daemon status``: reports the file as stale
- ``dfm run``: removes the stale file and falls back to ephemeral mode
- ``dfm daemon start``: removes the stale file and starts fresh


Monitoring
==========

Attach a live TUI to a running daemon::

    dfm daemon --monitor

Or start the daemon with the monitor attached::

    dfm daemon start --monitor

The monitor layout follows the style of Unix top:

- **Main area** (fills the screen): a scrolling table of currently
  running tasks showing the task name, assigned worker, host, and
  elapsed time. Below that, a "Recent Completions" table shows the
  last few finished tasks with pass/fail status.

- **Status bar** (fixed at the bottom, 4 lines): aggregate stats
  updated in real time:

  - Daemon uptime
  - Workers: total, busy, idle, pending
  - Tasks: currently running, total completed

Pressing q or Ctrl-C detaches the monitor; the daemon and its
workers continue running.