Starlark API

This page describes every global, member and function that kizz adds to Starlark, with its parameters, what it tracks as inputs and outputs, and its side effects. The kizz.md format describes the blocks that hold task programs, and the kizz.md guide explains how to write them.

Overview

A task program is Starlark, a small dialect of Python. The language itself, its types, statements, operators and built-in functions, is described by the Starlark language specification, not by this site. Kizz runs programs with the starlark crate, version 0.14.2, in its extended dialect. Besides the standard language, that dialect allows if and for statements at the top level of a file, keyword-only parameters after *, and type annotations.

The block's source plus every module it loads is the task program. Kizz parses the block with blank lines before it, so the line numbers in Starlark errors are the lines of the kizz.md. Errors in a module name the module's absolute path and its own lines.

Kizz adds these names to Starlark's standard built-in functions:

fail() and load() are part of the standard language, and their entries say what they do in a task. The other globals are those of the starlark crate's standard environment, such as len, sorted, getattr and type.

A program runs in one of two places:

Every file access that a program makes, in the block or while a module loads, is recorded against its task. Kizz uses these records to learn the order of tasks, to list the paths that a failed task looked for and did not find, and for the report of kizz run --output. They do not decide whether a command runs: each run() call has its own cache check. The program runs again in every pass, so its own reads always see current data.

run()

result = run(
    "cargo",
    "build",
    "--release",
    stdin=None,
    env=None,
    cwd=None,
    capture="split",
    stream=False,
    always=False,
    untraced=False,
    network=False,
    ignore=[],
    track=[],
    timestamps=False,
    machine=[],
    expect=None,
)

Runs one command, a step of the task, and returns its result. Commands are traced unless untraced = True is set. The positional arguments are the program and its arguments, and every other parameter is a keyword argument with a default. A positional argument can also be a list or tuple, which expands in place, as argv describes. run() works only in a task, not while a module loads.

Each call goes through these stages:

  1. Kizz builds the step's specification from the arguments and the task environment.
  2. It derives the step key, a BLAKE3 hash of the task program's hash and of the values that each entry below marks as part of the step key, plus the occurrence number: how many identical commands came earlier in the same run of the task.
  3. It checks the step's ignore and track patterns, and that a bare program name has a PATH in the step's environment. Either failure fails the task before kizz starts anything, in every pass, and the call counts as no command that ran. A step whose command met a tracer failure earlier in the same build fails again with that error, without a new run.
  4. It checks the cache. A step whose record holds a cached result, and whose recorded paths and named machine state are unchanged, is a cache hit: run() returns the recorded exit status and output, and nothing runs. An untraced or always step, every step of a cache=never task and every step under kizz run --no-cache get no cache check, so each runs again in every build or kizz run that reaches it.
  5. Otherwise it looks the program up and runs it. Unless untraced = True, it traces the command and records every path that the command's process tree touched, with the state of each path when the command finished. In a later pass of the same build, kizz returns the earlier result of an untraced or always step or of a step of a cache=never task instead, without running the command; kizz run runs its task once, with no later pass. It also returns a result that was not cached again, while no path that its run touched has changed. Such a result comes back with cached set, and a command that could not start fails the task again with the same error.
  6. It stores the step record under the step key, and returns the result.

The caching guide explains the cache check, and the caching controls list every option that changes it.

A step's record holds every parameter as the call gave it, with the positional arguments as their expanded strings, the program that ran, the exit status, the raw captured output, the paths with their states, the machine state, the network connections and the unknown system calls. Explanations use it to say what changed, and kizz why shows it.

argv

The positional arguments: the program, then its arguments. Each is a string, or a list or tuple whose elements are strings, lists or tuples.

Lists let a program build its command by logic, rather than write every argument literally. An optional argument can be a variable that holds the argument in a list when it is wanted, and [] when it is not:

release = ["--release"] if "release" in task.args else []
run("cargo", "build", "--locked", release, expect=[0])

Under kizz run with the argument release, this runs cargo build --locked --release, and otherwise cargo build --locked. In the same way, run("cargo", "test", task.args) passes the arguments of kizz run through, each as one argument, and with no arguments it runs cargo test. The same list may appear more than once: with opt = ["-v"], run("tool", opt, "x", opt) runs tool -v x -v.

Only strings, lists and tuples are accepted, and no other value is converted to a string. Any other value, such as an int, a bool, None, a dict, a range or a run_result, fails the task with an error that names its argument, its index path and its type, such as run(): argument 3 is of type int, not a string, list or tuple for run("head", "-n", 5), or run(): argument 2 at [1][0] is of type NoneType, not a string, list or tuple for run("p", ["a", [None]]). Arguments are counted from 1 among the positional arguments, and the index path is Starlark indexing into that argument, so argument 2 at [1][0] names x[1][0], where x is the second positional argument. The type is the name that Starlark's type() gives; str() converts a number when one is wanted, as in str(5).

A list or tuple that contains itself, directly or through other lists or tuples, fails the task where it is met again, such as run(): argument 2 at [1] is a list that contains itself for a = ["x"], a.append(a) and run("p", a); for a tuple, the message says tuple.

The expansion visits at most 1,000,000 values, counting every string, list and tuple, the positional arguments included. Arguments that would expand further, such as a = ["x"] followed by twenty-one rounds of a = [a, a], fail the task with run(): the arguments expand to more than 1000000 strings, lists and tuples.

These checks come in the order of the arguments, so run([], 5) fails on the type of argument 2, not as a call without a program. A call whose arguments fail to expand starts no command and stores no record.

stdin

The text for the command's standard input.

env

Variables for this one command, applied to the task environment.

The env global builds the task environment, which starts empty.

cwd

The command's working directory.

capture

How the command's output is captured.

stream

Whether the command's output is also shown as it comes.

always

Whether the command runs in every build that reaches it, even when its cache entry is valid.

A task marked cache=never runs every command as if it had always = True, and so does kizz run --no-cache.

untraced

Run a command that needs to use ptrace itself, such as kizz's own tracer tests. Linux permits only one ptrace tracer per process.

The direct child determines the exit status and leads a private process group. Forwarded signals and cancellation reach that group; cancellation escalates to a kill after the normal grace period. Remaining group members are killed when the direct child ends. Commands must keep descendants in that group and wait for their work before exiting. Only the direct child is configured to die if kizz dies abruptly. Other descendants can survive, including those that stayed in the original group, because no group cleanup then runs. Cancellation escalation kills the original group and the direct child, even if it changed groups. After the direct child ends, stdin closes and output is drained for at most one second or until cancellation. Remaining open output pipes are closed, and the report marks output as possibly cut short. Explicitly disabling tracing produces no incomplete-trace warning; the report still records that tracing was disabled. Use this explicit exception only when tracing prevents a command from working.

network

Acknowledge expected network or Unix socket activity, such as a tool downloading dependencies.

ignore

Paths that are not recorded as dependencies of this step.

Starlark variables make reusable sets of patterns:

GO_CACHE = ["~/.cache/go-build/**"]

run("go", "build", "./...", ignore=GO_CACHE, expect=[0])

track

Paths under /proc and /sys, or that symbolic links lead there, whose content counts as an input of this step.

run("./configure", track=["/proc/cpuinfo"], expect=[0])

timestamps

Whether a new timestamp of a used file or directory is a change.

machine

The parts of the machine state that count as an input of this step.

The host name, the user's credentials, the umask, the machine architecture and the CPU model are not per step: they are the cache's identity, which the caching guide describes.

expect

The exit codes that are acceptable, so that any other exit fails the task.

error: kizz.md:33 (Expect) failed:
  command `sh -c echo oops; exit 2`: exit code 2, but `expect` accepts exit code 0 or 1
  --- stdout ---
  oops
  --- stderr ---
  Traceback (most recent call last):
    * kizz.md:35, in <module>
        run("sh", "-c", "echo oops; exit 2", expect = [0, 1])
  error: run(): command `sh -c echo oops; exit 2`: exit code 2, which `expect` does not accept
    --> kizz.md:35:1
     |
  35 | run("sh", "-c", "echo oops; exit 2", expect = [0, 1])
     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     |

Program lookup

Failures

These fail the task at the run() call, with a Starlark error and the stack:

A program that is not found or not executable is recorded like any other step, with every location checked, so the step fails again from the cache until a checked location changes, and it runs again when the program appears. A command that cannot start for another reason, such as one whose cwd does not exist, has a result that is not cached. Within the build it runs again only when a path that it touched changes, and a later build runs it again. A bare program name without PATH and an invalid pattern fail the task before kizz starts anything, in every pass of a build. Kizz prints no ran, run or cached line for such a call, and counts it neither as a command that ran nor as a cached one. A tracer failure counts as a command that ran and leaves no record, and no later pass of the same build runs that command again.

A program that is not found names every location checked:

  error: run(): program `tool` was not found in the task environment's PATH; checked /opt/bin/tool, /opt/other/tool

Commands that finished before a failure keep their cache entries.

A program that nests values so deeply that kizz's stack overflows stops kizz itself, not only its task. Starlark's garbage collector copies nested values recursively, so a list wrapped in a list thousands of times can overflow the stack between two top-level statements. Kizz catches the overflow, prints an error that names the task, deletes the lock file and exits with 2:

error: the program of task kizz.md:3 (Deep) overflowed the stack, so kizz stopped; its values are probably nested too deeply

The cache stays as a crash leaves it, and the next build runs normally.

The result, run_result

run() returns a value of the Starlark type run_result, whose attributes depend on the capture mode: rc, signal, stdout and stderr, and cached, with "split"; rc, signal, output and cached, with "merged". Reading an attribute that the mode does not have is a Starlark error.

Reading an attribute tracks nothing. Reading rc or signal checks the result, which is its one side effect.

rc

The exit code, an integer, or None when a signal killed the command. It takes no parameters and tracks nothing. Reading it, as an attribute or with getattr, checks the result, which is its one side effect.

signal

The number of the signal that killed the command, an integer, or None when it exited. It takes no parameters and tracks nothing. Reading it, as an attribute or with getattr, checks the result, which is its one side effect.

stdout

The command's standard output, a string, with capture = "split". The bytes are decoded as UTF-8, with invalid bytes replaced by U+FFFD; the cache keeps the raw bytes. It takes no parameters, tracks nothing and has no side effect: reading it does not check the result.

stderr

The command's standard error, a string, with capture = "split", decoded as stdout is. It takes no parameters, tracks nothing and has no side effect: reading it does not check the result.

output

Both output streams in the order they were written, a string, with capture = "merged", decoded as stdout is. It takes no parameters, tracks nothing and has no side effect: reading it does not check the result.

cached

Whether kizz returned a recorded result instead of running the command for this call: a bool. It is True for a cache hit, and for a result that the same build returns again, such as that of an always step in a later pass. It takes no parameters, tracks nothing and has no side effect: reading it does not check the result.

Checking a result

Without expect, a non-zero exit or a signal does not fail the task at its run() call: the program decides what it means. A step's exit is unchecked when the command exited with a non-zero code or was killed by a signal, its run() had no expect, and the program, by the time it ended, never checked the result. The program checks a result when it:

hasattr, dir(), printing the result, and reading stdout, stderr, output or cached do not check it. A result from the cache counts like any other.

Once a build settles, kizz warns about each unchecked exit in the results that stand, those of each task's latest run, and repeats the warning in the summary. The warning names the task, the command and how it ended, and says that the program never read rc or signal.

kizz run warns once the run ends. kizz build --fail-unchecked and kizz run --fail-unchecked make each one fail its task instead.

env

Reads the caller environment and dotenv files, and changes the task environment.

env.inherit("PATH", "HOME")
env.load_file(".env", defaults={"GOFLAGS": "-trimpath"}, required=False)
env.set("CGO_ENABLED", "0")
run("go", "build", "./...", env={"GOFLAGS": None}, expect=[0])

env.caller()

env.caller(name, default = None) returns the value of the variable name in the caller environment, or default when it is unset.

env.inherit()

env.inherit(name, ...) copies the named variables from the caller environment into the task environment.

env.load_file()

env.load_file(path, defaults = {}, required = True) reads a dotenv file, fills in defaults for the variables it does not set, adds the result to the task environment, and returns it.

  error: env.load_file(): /work/bad.env:2: expected NAME=value, a comment starting with #, or a blank line, not `not an assignment`

env.read()

env.read(path) reads a dotenv file into a dict, without changing the task environment.

env.set()

env.set(name, value) sets the variable name of the task environment to value.

env.update()

env.update(values) sets every variable of the dict values in the task environment.

env.unset()

env.unset(name) removes the variable name from the task environment, if it is set.

env.get()

env.get(name, default = None) returns the value of the variable name in the task environment, or default when it is unset.

env.dict()

env.dict() returns the task environment as a new dict.

env.clear()

env.clear() empties the task environment.

Dotenv files

env.load_file() and env.read() read the same format:

Any other line is a syntax error that names its line, and it fails the task. Bytes that are not valid UTF-8 are replaced before the file is parsed.

fs

Reads and writes files, and inspects the file system.

fs.read()

fs.read(path) returns the text of a file.

fs.write()

fs.write(path, text) writes text to a file, creating its parent directories, unless the file already holds exactly text.

fs.read_json()

fs.read_json(path) returns the JSON value in a file as a Starlark value.

fs.write_json()

fs.write_json(path, value, indent = 2) writes value to a file as stable JSON, unless the file already holds exactly that text.

fs.write_json("out/data.json", {"b": [1, 2], "a": True})

This writes:

{
  "a": true,
  "b": [
    1,
    2
  ]
}

fs.exists()

fs.exists(path) returns whether a path exists, following symbolic links.

fs.list()

fs.list(path = ".") returns the names of the entries of a directory, sorted.

fs.glob()

fs.glob(pattern) returns the paths that match a glob pattern, sorted.

for manifest in fs.glob("services/*/package.json"):
    print(manifest)

task

Describes the running task. Its members are values, not functions, and they cannot be changed. task is also available while a module loads, and describes the task that loads it.

None of them is part of the task program's hash: a value reaches a step key only through what the program passes to a command.

task.dir

The task directory, the absolute directory of the task's kizz.md, as a string. It is the directory that relative paths of run(), env and fs are resolved against. It takes no parameters, tracks nothing and has no side effect.

task.file

The absolute path of the task's kizz.md, as a string. It takes no parameters, tracks nothing and has no side effect.

task.line

The line of the block's opening fence in its kizz.md, counting from 1, as an int. It takes no parameters, tracks nothing and has no side effect.

task.name

The name that the block's name= option gives, as a string, or None when it has none. Before tasks had names, task.name held the heading, which is now task.heading. It takes no parameters, tracks nothing and has no side effect.

task.heading

The task's heading, the nearest Markdown heading above the block, as plain text, or None when no heading is above it. It takes no parameters, tracks nothing and has no side effect.

task.targets

The targets on the block, as a list of strings, in the order written and without repeats. It is an empty list for a task that has only a name. It takes no parameters, tracks nothing and has no side effect.

task.selected

The requested targets that selected the task, as a list of strings, in the order of the target expression. This is the intersection of the requested targets and task.targets. For kizz build fix+check+ci, a task tagged fix ci sees ["fix", "ci"] and a task tagged only ci sees ["ci"]. A task that supports both modes can branch on "fix" in task.selected. Under kizz run it is an empty list, because no target selected the task. It takes no parameters, tracks nothing and has no side effect.

task.args

The task's arguments, the words after -- of kizz run, as a list of strings. Under kizz build it is an empty list. The arguments are not part of the task program's hash: they reach commands only through what the program does with them. An argument that the program passes to a command is part of that command's step key, so a new value gives it another step key. It takes no parameters, tracks nothing and has no side effect.

who = task.args[0] if task.args else "world"
run("echo", "hello,", who, expect=[0])

Because run() expands a list in place, run("cargo", "test", task.args) passes every argument to the command, each as one argument, and adds nothing when task.args is empty.

json

The json module of the starlark crate, which kizz adds to the globals. It works in a task and while a module loads, tracks nothing, and has no side effect apart from checking the results it encodes.

json.encode()

json.encode(value) returns the compact JSON text of a value.

json.decode()

json.decode(text) returns the Starlark value of JSON text.

print()

print(value, ...) prints its arguments as one line of the task's output.

show: run_result(rc=0, signal=None, output="out\nerr\n", cached=False)

See the Starlark specification for the standard function.

fail()

fail(value, ...) fails the task.

error: kizz.md:27 (Fail) failed:
  Traceback (most recent call last):
    * kizz.md:28, in <module>
        fail("the version is", 3, None)
  error: fail: the version is 3 None
    --> kizz.md:28:1
     |
  28 | fail("the version is", 3, None)
     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     |

See the Starlark specification for the standard function.

load()

load("path/to/lib.star", "name", alias = "name") imports names from a Starlark module file. It is a statement of the language, which the Starlark specification describes.

load("../build/common.star", "go_build")

env.inherit("PATH", "HOME")
go_build("./cmd/server", output="dist/server", version="1.0.0")