Command reference

dv_flow_mgr

usage: dfm [-h] [--log-level {NONE,INFO,DEBUG}] {graph,run,show,util} ...

Named Arguments

--log-level

Possible choices: NONE, INFO, DEBUG

Configures debug level [INFO, DEBUG]

Sub-commands

graph

Generates the graph of a task

dfm graph [-h] [-f FORMAT] [--root ROOT] [-o OUTPUT] [task]

Positional Arguments

task

task to graph

Named Arguments

-f, --format

Specifies the output format

Default: 'dot'

--root

Specifies the root directory for the flow

-o, --output

Specifies the output file

Default: '-'

run

run a flow

dfm run [-h] [-j J] [--clean] [--root ROOT] [tasks ...]

Positional Arguments

tasks

tasks to run

Named Arguments

-j

Specifies degree of parallelism. Uses all cores by default

Default: -1

--clean

Cleans the rundir before running

Default: False

--root

Specifies the root directory for the flow

show

Display information about a task or tasks

dfm show [-h] [-a] [-v] [task]

Positional Arguments

task

task to show

Named Arguments

-a, --all

Shows all tasks required for the subject to run

Default: False

-v, --verbose

Shows additional information about tasks

Default: False

util

Internal utility command

dfm util [-h] cmd ...

Positional Arguments

cmd
args

Output Directory Structure

DV Flow Manager creates an output directory structure that mirrors the task graph being executed. Each top-level task has a directory within the run directory. Compound tasks have a nested directory structure.

There are two top-level directory that always exist:

  • cache - Stores task memento data and other cross-run artifacts

  • log - Stores execution trace and log files

Each task directory contains some standard files:

  • <task_name>.exec_data.json - Information about the task inputs, outputs, and executed commands.

  • logfiles - Command-specific log files

Viewing Task Execution Data

After a run has completed, the log directory will contain a JSON-formatted execution trace file named <root_task>.trace.json. This file is formatted in Google Event Trace Format, and can be processed by tools from the Perfetto project.

An execution is shown in the Perfetto UI below. In addition to seeing information about how tasks executed with respect to each other, data about individual tasks can be seen.

imgs/perfetto_trace_view.png