Developing Tasks¶

The Using Tasks chapter describes how to customize existing tasks by specifying parameter values and using compound tasks to compose tasks from a collection of sub-tasks. When adding a new tool or capability, you will often need finer-grained control: a task that runs a specific command, generates files, or invokes a tool. DV Flow lets you implement such a task directly in the flow file.

The recommended starting point is a shell-script task – you put the commands in the task’s run body and DV Flow handles scheduling, inputs, outputs, and up-to-date tracking. When a script grows unwieldy – when you need rich control flow, typed data manipulation, or in-process access to DV Flow APIs – you can graduate the same task to a Python implementation.

Task Execution¶

A task selects its implementation with two parameters:

run – the body of the implementation (shell commands, or Python code / a Python entry-point reference).
shell – the interpreter for run. bash (the default), shell, csh, and tcsh run run as a shell script; pytask runs it as Python.

If shell is omitted, the task is a shell-script task executed with bash.

Shell-Script Tasks¶

A shell-script task puts one or more shell commands in the run body. This is the most direct way to wrap a tool or generate files, and it is the recommended place to start.

package:
  name: my_tool
  tasks:
  - name: gen_rtl
    with:
      top:
        type: str
        value: top
    run: |
      mkdir -p rtl
      echo "module ${{ top }}(); endmodule" > rtl/${{ top }}.v
      echo "Generated rtl/${{ top }}.v"

The run body is written to a script and executed. ${{ }} expressions are substituted before the script runs, so ${{ top }} above expands to the value of the task’s top parameter. Other useful expressions include ${{ rundir }} (the task’s run directory) and ${{ this.<param> }}.

You can select a different shell explicitly:

- name: report
  shell: bash
  run: |
    echo "Building in ${{ rundir }}"

Exchanging data with the dataflow graph¶

Shell tasks exchange data with the rest of the graph through a GitHub-Actions-style contract: the runner passes inputs as ``DFM_*`` environment variables (parameters via DFM_PARAM_*/DFM_PARAMS, consumed inputs via DFM_INPUTS, the run directory via DFM_RUNDIR) and collects outputs from append-only files the script writes – produced filesets (DFM_OUTPUT), environment/PATH additions (DFM_ENV / DFM_PATH), diagnostics (DFM_MARKERS), and the memento used for up-to-date checks (DFM_MEMENTO_OUT). The dfm-out helper writes these output files for you.

See Script ↔ Dataflow I/O for the full input/output contract and the dfm-out reference.

Python Task Implementation¶

When a shell script becomes unwieldy – you need loops and conditionals over structured data, want to manipulate typed inputs/outputs directly, or need in-process access to DV Flow services – implement the task in Python instead. Set shell: pytask and provide the Python code inline, or reference an external Python entry point.

External Pytask¶

A pytask implementation for a task is provided by a Python async method that accepts input parameters from the DV Flow runtime system and returns data to the system. When the pytask implementation is external, the run parameter specifies the name of the Python method.

package:
  name: my_tool
  tasks:
  - name: my_task
    uses: my_package.MyTask
    shell: pytask
    run: my_package.my_module.MyTask
    with:
      msg:
        type: str

The task definition above specifies that a pytask implementation for the task is provided by a Python method named my_package.my_module.MyTask.

async def MyTask(ctxt, input):
    print("Message: %s" % input.params.msg)

See the Python Task API documentation for more information about the Python API available to task implementations.

This “external” implementation makes the most sense when the task implementation is moderately complex or lengthy.

Inline Pytask¶

When the task implementation is simple, the code can be in-lined within the YAML.

package:
  name: my_tool
  tasks:
  - name: my_task
    uses: my_package.MyTask
    with:
      msg:
        type: str
    shell: pytask
    run: |
      print("Message: %s" % input.params.msg)

When this task is executed, the body of the run entry will be evaluated as the body of an async Python method that has ctxt, and input parameters.

Task-Graph Expansion¶

Sometimes build flows need to run multiple variations of the same core step. For example, we may wish to run multiple UVM tests that only vary in the input arguments. The matrix strategy can work well in these cases.

package:
  name: my_pkg

  tasks:
  - name: SayHi
    strategy:
      matrix:
        who: ["Adam", "Mary", "Joe"]
    body:
      - name: Output
        uses: std.Message
        with:
          msg: "Hello ${{ matrix.who }}!"

The matrix strategy is only valid on compound tasks. The body tasks are evaluated once for each combination of matrix variables. Body-task parameters can reference the matrix variables.

In this case, we would expect the SayHi task to look like this when expanded:

        flowchart TD
  A[SayHi.in]
  B[Hello Adam!]
  C[Hello Mary!]
  D[Hello Joe!]
  E[SayHi]
  A --> B
  A --> C
  A --> D
  B --> E
  C --> E
  D --> E

Task-Graph Generation¶

It is sometimes useful to generate task graphs programmatically instead of capturing them manually or generating them textually in YAML. A generate strategy can be provided to algorithmically define a task subgraph.

Note that generation is done statically as part of graph elaboration. This means that the generated graph structure may only depend on values, such as parameter values, that are known during elaboration. The graph structure cannot be created using data conveyed as dataflow between tasks.

package:
  name: my_pkg

  tasks:
  - name: SayHi
    with:
      count:
        type: int
        value: 1
    strategy:
      generate: my_pkg.my_mod.GenGraph

The generate strategy specifies that the containing task will be a compound task whose sub-tasks are provided by the specified generator. As with other task implementations, the generator code can be specified externally in a Python module or inline.

def GenGraph(ctxt, input):
    count = input.params.count
    for i in range(count):
        ctxt.addTask(ctxt.mkTaskNode(
            "std.Message", with={"count": 1})
            name=ctxt.mkName("SayHi%d" % i),
            msg="Hello World% %d!" % (i+1)))

See the Python Task API documentation for more information about the Python task-graph generation API.

Task-Graph Generation and Error Handling¶

Dynamically-generated subgraphs support the same max_failures control as static compound tasks. Pass max_failures to run_subgraph() to control how many subtask failures are tolerated before remaining independent siblings are skipped:

async def run_tests(ctxt, input):
    # Build test task nodes dynamically …
    tasks = [build_test_node(ctxt, seed) for seed in seeds]

    # Run all; failures do not abort siblings.
    await ctxt.run_subgraph(tasks, max_failures=-1)
    return TaskDataResult(status=0, output=[])

See Error Handling for the full max_failures semantics.

For larger Python implementations – the class-based PyTask API and the PyPkg package factory – see Python Tasks.

Template Tasks¶

A template task defers expansion of its run expression from load time to graph-build time. This is useful for reusable building blocks whose run expression references variables that only exist in the use context (matrix variables, compound parameters, package parameters of the consuming package, etc.).

Declaring a Template Task¶

Add template: true to the task definition:

package:
  name: sim_pkg

  tasks:
  - name: CompileStub
    template: true
    shell: bash
    run: "echo Skipping compile for ${{ matrix.variant }}"
    passthrough: all
    consumes: none

The run string is stored verbatim at load time. When the task is instantiated (via uses: or as an override replacement), the graph builder expands ${{ }} expressions using the instantiation context.

Using a Template Task¶

A template task is consumed through uses:, just like any other task:

tasks:
- name: MyCompile
  uses: sim_pkg.CompileStub

At graph-build time, ${{ matrix.variant }} (or whichever variables appear in run) are resolved against the current context.

Template tasks work naturally inside strategy.matrix:

tasks:
- name: StubMatrix
  strategy:
    matrix:
      variant: [rtl, gate]
  body:
  - name: Step
    uses: sim_pkg.CompileStub

Each matrix cell gets its own expansion, so ${{ matrix.variant }} resolves to rtl and gate respectively.

Constraints¶

A template task cannot be invoked directly from the CLI or as a top-level entry point. Doing so raises an error.
template: true and override: are mutually exclusive on the same task definition.
Parameter definitions (with:) are unaffected – they are already expanded lazily at graph-build time regardless of the template flag.

When to Use Templates¶

Use template: true when:

The run expression references variables that are not available at load time (e.g. ${{ matrix.variant }}, ${{ this.some_param }}).
You are building a reusable task that will be consumed by multiple packages with different parameter contexts.
You need the same task definition to produce different shell commands depending on where it is instantiated.