Flow-Spec Reference

Conventions

The reference documentation uses a few conventions when specifying the capabilities of various aspects of the specification:

• refstring - A string where parameter and expression references (ie ${{ … }}) are expanded.

• string - A literal string

Packages

A package file defines content under a package root element. The name element is required. Other elements are optional.

Minimal package definition

package:
  name: my_package

An outline of the package schema is shown below. Please see the following sections for more information about key items:

• Datasets

• Tasks

Schema:	#/defs/PackageDef
Item	Type	Description
name	string	Name of the package
desc	string	Short description of the package
doc	string	Documentation of the package content
uses	string	Name of the base package
with	list[refstring | ParamDef]	Package parameters
imports	list[refstring]	External packages to load
tasks	list[TaskDef]	Task definitions
datasets	list[DatasetDef]	Dataset definitions
fragments	list[refstring]	Package fragment files

A package file defines content under a package root element. The name element is required. Other elements are optional.

Minimal package definition

package:
  name: my_package

Uses

A package inherits from a base package via the uses element. All elements from the base package become elements of the inheriting package.

Package inheritance

# my_base_pkg.yaml
package:
  name: my_base_pkg
  tasks:
  - name: my_t1

# my_ext_pkg.yaml
package:
  name: my_ext_pkg
  uses: my_base_pkg
  imports:
  - my_base_pkg.yml
  tasks:
  - name: my_t2

In the example above, my_ext_pkg contains two tasks: my_t1 and my_t2.

With

A package declares and configures package-local parameters using the with section.

Package parameter declaration

package:
  name: my_package
  with:
  - name: p1
    type: str
    value: abc

For more information about declaring and using parameters, see the Parameters section.

Imports

Packages are explicitly loaded from a file using the imports section. Each entry in this section is a path to a package file. The path may be absolute or relative. Relative paths are resolved:

• Relative to the directory containing the current file

• Relative to the directory containing the root package file

Variable references are supported.

Package import with an environment-rooted path

package:
  name: my_package
  imports:
  - ${{ env.ANCHOR_my_uvc }}/flow.yaml

Package import with a relative path

package:
  name: my_package
  imports:
  - import/my_ip/flow.yaml

Package Fragments

Package fragment files specify additional content. Most package elements can be declared in a fragment, with the exception of the package name, uses (base package), and with (package parameters).

Package fragment

fragment:
  imports:
  - ../my_file.yaml
  tasks:
  - name: T1
    # ...

Schema:	#/defs/FragmentDef
Item	Type	Description
imports	list[refstring]	External packages to load
tasks	list[TaskDef]	Task definitions
datasets	list[DatasetDef]	Dataset definitions
fragments	list[string]	Package fragment files

Task Definition

RundirE

type	string
enum	unique, inherit

TaskDef

Holds definition information (ie the YAML view) for a task
type	object
properties
name	Task Name
	The name of the task
	default	null
	anyOf	type	string
		type	null
root	Root Task Name
	The name of the task (marked as root scope)
	default	null
	anyOf	type	string
		type	null
export	Export Task Name
	The name of the task (marked as export scope)
	default	null
	anyOf	type	string
		type	null
local	Local Task Name
	The name of the task (marked as local scope)
	default	null
	anyOf	type	string
		type	null
override	Overide Name
	The name of the task to override
	default	null
	anyOf	type	string
		type	null
uses	Base type
	Task from which this task is derived
	type	string
	default	null
scope	Task visibility scope
	Visibility scope: ‘root’ (executable), ‘export’ (visible outside package), ‘local’ (fragment-only)
	default	null
	anyOf	type	string
		type	array
		items	type	string
		type	null
body	Body
	Sub-tasks
	type	array
	items	TaskDef
iff	Task enable condition
	Condition that must be true for this task to run
	default	null
	anyOf	type	string
		type	boolean
pytask	Pytask
	Python-based implementation (deprecated)
	type	string
	default	null
run	Run
	Shell-based implementation
	type	string
	default	null
shell	Shell
	Shell to use for shell-based implementation
	type	string
	default	bash
strategy	default	null
	StrategyDef
desc	Task description
	Short description of the task’s purpose
	type	string
	default
doc	Task documentation
	Full documentation of the task
	type	string
	default
needs	Needs
	List of tasks that this task depends on
	type	array
	items	type	string
feeds	Feeds
	List of tasks that depend on this task (inverse of needs)
	type	array
	items	type	string
with	With
	Parameters for the task
	type	object
	additionalProperties	anyOf	type	string
			type	array
			items
			type	integer
			type	boolean
			type	object
			additionalProperties	True
rundir	Specifies handling of this tasks’s run directory
	default	unique
	RundirE
passthrough	Passthrough
	Specifies whether this task should pass its inputs to its output
	default	null
	anyOf	#/defs/PassthroughE
		type	array
		items
		type	null
consumes	Consumes
	Specifies matching patterns for parameter sets that this task consumes
	default	null
	anyOf	#/defs/ConsumesE
		type	array
		items
		type	null
uptodate	Uptodate
	Up-to-date check: false=always run, string=Python method, None=use default check
	default	null
	anyOf	type	boolean
		type	string
		type	null
cache	Cache
	Cache configuration. True=enabled with defaults, False=disabled, CacheDef=custom config
	default	null
	anyOf	#/defs/CacheDef
		type	boolean
		type	null
tags	Tags
	Tags as type references with optional parameter overrides
	type	array
	items	anyOf	type	string
			type	object
			additionalProperties	True
additionalProperties	False

PackageImportSpec

type	object
properties
from	From
	Package identifier or file path to import
	type	string
	default	null
as	As
	Alias name for the imported package
	type	string
	default	null
config	Config
	Configuration name to apply when importing
	type	string
	default	null
with	With
	Parameter overrides to apply to the imported package
	type	object
	additionalProperties	True

TypeDef

type	object
properties
name	Name
	Name of the data type
	type	string
uses	Uses
	Base type to inherit from. Supports type extension/specialization
	type	string
	default	null
doc	Doc
	Documentation for this type definition
	type	string
	default	null
with	With
	Parameters for this type. Can be simple values or full parameter definitions
	type	object
	additionalProperties	anyOf	type	string
			ParamDef
tags	Tags
	Tags as type references with optional parameter overrides
	type	array
	items	anyOf	type	string
			type	object
			additionalProperties	True

StrategyDef

type	object
properties
chain	Chain
	Enable chain strategy: run body tasks sequentially with each consuming output of previous task
	default	null
	anyOf	type	boolean
		type	null
generate	Enable generate strategy: dynamically create tasks by running a shell command that outputs task definitions
	default	null
	anyOf	#/defs/GenerateSpec
		type	null
matrix	Matrix
	Matrix of parameter values to explore. Creates one task instance per combination of values
	default	null
	anyOf	type	object
		additionalProperties	type	array
			items
		type	null
body	Body
	Body tasks for strategy execution. Used with chain and matrix strategies
	type	array
	items	TaskDef

ListType

type	object
properties
item	Item
	anyOf	type	string

MapType

type	object
properties
key	Key
	anyOf	type	string
val	Val
	anyOf	type	string

ComplexType

type	object
properties
list	default	null
	anyOf	ListType
		type	null
map	default	null
	anyOf	MapType
		type	null

ParamDef

type	object
properties
doc	Doc
	Full documentation for this parameter
	type	string
	default	null
desc	Desc
	Short description of this parameter
	type	string
	default	null
type	Type
	Parameter type (e.g., ‘str’, ‘int’, ‘bool’, ‘list’, ‘map’, or a complex type definition)
	default	null
	anyOf	type	string
		ComplexType
value	Value
	Default value for this parameter
	default	null
	anyOf	type	null
append	Append
	Value to append to list-type parameters
	default	null
	anyOf	type	null
prepend	Prepend
	Value to prepend to list-type parameters
	default	null
	anyOf	type	null
path-append	Path-Append
	Path to append to path-type parameters (OS-specific separator)
	default	null
	anyOf	type	null
path-prepend	Path-Prepend
	Path to prepend to path-type parameters (OS-specific separator)
	default	null
	anyOf	type	null