Expand description
The engine side of the step executor.
An Engine holds what every task of one build shares: the tracer, the store,
where live output goes, the cancellation flag, the caller’s home directory, the
in-memory cache of file states and hashes, the results of the always steps
that already ran, the results that were not cached, and the tracer failures of
the build. For each run of a task program, Engine::task makes a
TaskSteps, the StepExecutor that the runtime hands each run() call to.
For each step it:
- derives the step key, counting identical commands earlier in the run;
- compiles the step’s ignore and
trackpatterns, and checks that a bare program name has aPATHin the step’s environment; - returns the tracer failure that the build keeps for the step key, if any;
- decides whether the step runs, with the cache check;
- on a cache hit, returns the recorded exit status and raw output, with
cachedset, storing new metadata the check found; - otherwise looks the program up through the
PATHof the step’s environment, recording every location checked, queries the machine state that the command starts with, through the tracer’s machine query, notes that the step ran, and runs the command through the tracer; - drops the cached hash of every path the step wrote, and records the state of every
touched path, concurrently, applying the step’s ignore and
trackpatterns; and - stores the step record, with the machine state, under the step key; a result that is not cached replaces any older record with a record marked as not cached.
The cache check (“Cache check”): with no record for the key, the step runs, and
the reason is that it never ran. With a record, every recorded observation is
compared with the file system by check; the step is a
hit only when nothing changed, and otherwise it runs, with every difference as
the reason.
Machine state (“Machine state”): a cache hit queries no machine state to record.
When the record names parts of the machine state, with run(..., machine = [...]), the check queries the current state for the record’s working directory
and compares each value of those parts with the recorded one; a difference makes
the step run, and the reason names each changed value, old and new. Each step key
is compared at most once per build, and not after its step ran in the build, so a
step that names a part that changes all the time, such as memory, runs at most
once per build and the build still settles.
A step marked always runs at most once per build, when a pass first reaches it:
the first time its key comes up in the build, normally in the first pass, and it
is recorded like any other step. Every later time in the same build, whatever
changed, its first result is returned again without running (“Commands that
always run”). A key first reached in a later pass runs once then. Every step of a
task marked cache=never runs the same way, as if it were marked always, and
its reason names the task option. With Engine::with_no_cache (kizz run --no-cache), every step runs the same way too, without a cache check, and is
still recorded; its reason names the option, unless the step is marked always,
its task is marked cache=never, or it never ran, whose reasons come first. The
option never makes a step cached.
Step events (“Running one task with kizz run”): a run made with
TaskSteps::with_events tells its StepEvents when a step starts to run,
before its command starts and so before any of its output, and when a step comes
from the cache, with the result that the step returns. With
Engine::with_echo_every_step, every command’s output is echoed live, as if
every run() had stream = True; the option is in no step key or record.
Observation reports (“Running one task with kizz run”): with
Engine::with_report (kizz run --output), every request asks the tracer for
the process tree, and each run of a task program keeps, in the order its steps
came, the CommandInput of each step (TaskSteps::commands): a step that
ran, or whose program kizz did not find, with its outcome, its tree, and every
location that the lookup checked with its outcome, named through the tracer’s
error-name query; and a cache hit as cached, with its recorded outcome and no
observations. Without it, as in every build, no request asks for the tree and
nothing more is kept. The option is in no step key or record.
Results that are not cached (“Results that are not cached”): a command killed by
a signal, every run that returns once cancellation has been requested, whether a
signal ended it or it exited with a code after it handled the interrupt, a run
that the tracer reports as incompletely traced, a run in which a file changed
after the command first read it or ran it, and a command that could not start
for a reason other than a missing or non-executable program. A run that made a
system call that the tracer does not know is not cached either, unless the build
caches such results (kizz build --cache-unknown-syscalls, see
Engine::with_cache_unknown_system_calls); the tracer reports only the
numbers of such calls, and the record lists them either way. After it records
the states, the engine compares the first-read metadata that the tracer noted
with them (see changed_after_read); a change, when no other reason applies,
is the reason, and lists each changed file with whether the command wrote it.
Their record is marked with why, which makes it a record that is never a cache
hit and replaces any older record of the step key, so no older result is
replayed. The engine also keeps the paths such a command touched, with their
states, in memory for the rest of the build: when the step comes up again, it
runs again only if one of them changed, and otherwise its result is returned
again without running. In a later build, the marked record makes the step run
again, with that stated as the reason.
A run’s connections to IPv4 and IPv6 addresses and to Unix sockets, which the
tracer reports, never keep its result out of the cache: the record lists them,
and the engine keeps each run that made any for the build’s warnings (see
Engine::connections). A cache hit runs nothing, so it adds no warning.
Unchecked steps (“Exit codes and failure”): with Engine::with_fail_unchecked,
a task run that leaves a non-zero exit unchecked fails when its program ends.
Which unchecked steps warn is the build’s to say, from the results that stand
when it settles (see BuildReport::unchecked).
A missing or non-executable program fails the task and is recorded like any other failure, with every path checked and the failure message, which a cache hit fails with again.
Commands that ran (“Order of execution”): a step is noted as ran, with its
decision, its warning for a repeated reason and its step event, only once kizz is
about to record that its program is missing or not executable, or to ask the
tracer to run its command: after the lookup and the query of the machine state.
A run() that fails before that point has no decision, so it is neither a
command that ran nor a cached one, and it does no work for its pass. A bare
program name without PATH in the step’s environment and a malformed pattern
fail the task without a record; kizz checks both from the step’s own values
before the cache check, so every pass fails such a call the same way, and kizz
never attempts it. The other failures before that point are failures of kizz’s
own state: the store’s reads of the cache check and of the explanation, and the
locks of the build’s shared state. They leave nothing in the build’s memories,
neither an always result nor a compared machine state, so a later pass
evaluates the call again, since that state can change. A tracer failure counts as
a command that ran, because the engine cannot tell whether the tracer started
anything, and it leaves no record; the engine keeps it by step key for the rest
of the build, so a later pass returns the same failure before the cache check,
without a decision and without asking the tracer again. A record that cannot be
stored belongs to a command that ran, and the engine does not keep it.
Explanations (“Explaining decisions”): before a step runs, the engine turns the
cache check’s reason into an Explanation (see explain), and
each step’s StepDecision is kept for the line kizz build prints. For a task
run made with Engine::task_for, which knows its TaskIdentity, the engine
also stores, with each step record it writes, a last-writer record for every path
the step wrote and the step-position record with the explanation; a cached step’s
position is stored when TaskSteps::finish stores the task record, which also
removes the positions the run no longer reached. Engine::program_wrote stores
a task program as the last writer of each file it changed through fs.write or
fs.write_json. A step that re-runs in consecutive builds for the same
self-inflicted reason adds a RepeatedWarning.
Owners (“Cleaning up the cache”): every step record carries the tasks that own
it, by TaskId. A run made with Engine::task_for adds
its task to the owners of the record it stores, in the transaction that stores
it, and keeps the owners of the record it replaces under the same key, a record
marked as not cached included; a run without a task identity
adds no owner and keeps the others. A step noted as cached, whether a hit on a
stored record or a result reused within the build, makes its task an owner of
the record under its key in the transaction of TaskSteps::finish that stores
its position; a hit that refreshes a record’s observations keeps its owners. So
after a build, every task that stores a step position for a key owns the record
under that key. A path reset and kizz cache clean use the owners (see
cleanup).
Shared state (“Kizz never panics”): the engine’s state that the threads of a
build share is locked with kizz_threads::lock_within, never with
Mutex::lock, which may panic when the calling thread holds the lock; a lock
that stays held for the engine’s lock patience, kizz_threads::LOCK_PATIENCE
unless Engine::with_lock_patience sets another, is an error of the step, or
of the call that reads the state. The state of one task run lives in Cells
that the private cells module changes, never in a RefCell, whose borrows
panic while the cell is borrowed.
Structs§
- Engine
- What every task of a build shares to run, check, and record its steps.
- NoStep
Events - Follows no step.
- Step
Connections - A run of a step that connected to IPv4 or IPv6 addresses or to Unix sockets, as the engine keeps it for the build’s warnings.
- Task
Steps - The step executor of one run of a task program.
- Unknown
Calls - A run of a step that made system calls that the tracer does not know, as the engine keeps it for the build’s warnings.
Traits§
- Step
Events - Follows the steps of a task run as they start or come from the cache, for the
live output of
kizz run.