Skip to main content

Module engine

Module engine 

Source
Expand description

The engine side of the step executor.

An Engine holds what every task of one build shares: the tracer, the store, where live output goes, the cancellation flag, the caller’s home directory, the in-memory cache of file states and hashes, the results of the always steps that already ran, the results that were not cached, and the tracer failures of the build. For each run of a task program, Engine::task makes a TaskSteps, the StepExecutor that the runtime hands each run() call to. For each step it:

  1. derives the step key, counting identical commands earlier in the run;
  2. compiles the step’s ignore and track patterns, and checks that a bare program name has a PATH in the step’s environment;
  3. returns the tracer failure that the build keeps for the step key, if any;
  4. decides whether the step runs, with the cache check;
  5. on a cache hit, returns the recorded exit status and raw output, with cached set, storing new metadata the check found;
  6. otherwise looks the program up through the PATH of the step’s environment, recording every location checked, queries the machine state that the command starts with, through the tracer’s machine query, notes that the step ran, and runs the command through the tracer;
  7. drops the cached hash of every path the step wrote, and records the state of every touched path, concurrently, applying the step’s ignore and track patterns; and
  8. stores the step record, with the machine state, under the step key; a result that is not cached replaces any older record with a record marked as not cached.

The cache check (“Cache check”): with no record for the key, the step runs, and the reason is that it never ran. With a record, every recorded observation is compared with the file system by check; the step is a hit only when nothing changed, and otherwise it runs, with every difference as the reason.

Machine state (“Machine state”): a cache hit queries no machine state to record. When the record names parts of the machine state, with run(..., machine = [...]), the check queries the current state for the record’s working directory and compares each value of those parts with the recorded one; a difference makes the step run, and the reason names each changed value, old and new. Each step key is compared at most once per build, and not after its step ran in the build, so a step that names a part that changes all the time, such as memory, runs at most once per build and the build still settles.

A step marked always runs at most once per build, when a pass first reaches it: the first time its key comes up in the build, normally in the first pass, and it is recorded like any other step. Every later time in the same build, whatever changed, its first result is returned again without running (“Commands that always run”). A key first reached in a later pass runs once then. Every step of a task marked cache=never runs the same way, as if it were marked always, and its reason names the task option. With Engine::with_no_cache (kizz run --no-cache), every step runs the same way too, without a cache check, and is still recorded; its reason names the option, unless the step is marked always, its task is marked cache=never, or it never ran, whose reasons come first. The option never makes a step cached.

Step events (“Running one task with kizz run”): a run made with TaskSteps::with_events tells its StepEvents when a step starts to run, before its command starts and so before any of its output, and when a step comes from the cache, with the result that the step returns. With Engine::with_echo_every_step, every command’s output is echoed live, as if every run() had stream = True; the option is in no step key or record.

Observation reports (“Running one task with kizz run”): with Engine::with_report (kizz run --output), every request asks the tracer for the process tree, and each run of a task program keeps, in the order its steps came, the CommandInput of each step (TaskSteps::commands): a step that ran, or whose program kizz did not find, with its outcome, its tree, and every location that the lookup checked with its outcome, named through the tracer’s error-name query; and a cache hit as cached, with its recorded outcome and no observations. Without it, as in every build, no request asks for the tree and nothing more is kept. The option is in no step key or record.

Results that are not cached (“Results that are not cached”): a command killed by a signal, every run that returns once cancellation has been requested, whether a signal ended it or it exited with a code after it handled the interrupt, a run that the tracer reports as incompletely traced, a run in which a file changed after the command first read it or ran it, and a command that could not start for a reason other than a missing or non-executable program. A run that made a system call that the tracer does not know is not cached either, unless the build caches such results (kizz build --cache-unknown-syscalls, see Engine::with_cache_unknown_system_calls); the tracer reports only the numbers of such calls, and the record lists them either way. After it records the states, the engine compares the first-read metadata that the tracer noted with them (see changed_after_read); a change, when no other reason applies, is the reason, and lists each changed file with whether the command wrote it. Their record is marked with why, which makes it a record that is never a cache hit and replaces any older record of the step key, so no older result is replayed. The engine also keeps the paths such a command touched, with their states, in memory for the rest of the build: when the step comes up again, it runs again only if one of them changed, and otherwise its result is returned again without running. In a later build, the marked record makes the step run again, with that stated as the reason.

A run’s connections to IPv4 and IPv6 addresses and to Unix sockets, which the tracer reports, never keep its result out of the cache: the record lists them, and the engine keeps each run that made any for the build’s warnings (see Engine::connections). A cache hit runs nothing, so it adds no warning.

Unchecked steps (“Exit codes and failure”): with Engine::with_fail_unchecked, a task run that leaves a non-zero exit unchecked fails when its program ends. Which unchecked steps warn is the build’s to say, from the results that stand when it settles (see BuildReport::unchecked).

A missing or non-executable program fails the task and is recorded like any other failure, with every path checked and the failure message, which a cache hit fails with again.

Commands that ran (“Order of execution”): a step is noted as ran, with its decision, its warning for a repeated reason and its step event, only once kizz is about to record that its program is missing or not executable, or to ask the tracer to run its command: after the lookup and the query of the machine state. A run() that fails before that point has no decision, so it is neither a command that ran nor a cached one, and it does no work for its pass. A bare program name without PATH in the step’s environment and a malformed pattern fail the task without a record; kizz checks both from the step’s own values before the cache check, so every pass fails such a call the same way, and kizz never attempts it. The other failures before that point are failures of kizz’s own state: the store’s reads of the cache check and of the explanation, and the locks of the build’s shared state. They leave nothing in the build’s memories, neither an always result nor a compared machine state, so a later pass evaluates the call again, since that state can change. A tracer failure counts as a command that ran, because the engine cannot tell whether the tracer started anything, and it leaves no record; the engine keeps it by step key for the rest of the build, so a later pass returns the same failure before the cache check, without a decision and without asking the tracer again. A record that cannot be stored belongs to a command that ran, and the engine does not keep it.

Explanations (“Explaining decisions”): before a step runs, the engine turns the cache check’s reason into an Explanation (see explain), and each step’s StepDecision is kept for the line kizz build prints. For a task run made with Engine::task_for, which knows its TaskIdentity, the engine also stores, with each step record it writes, a last-writer record for every path the step wrote and the step-position record with the explanation; a cached step’s position is stored when TaskSteps::finish stores the task record, which also removes the positions the run no longer reached. Engine::program_wrote stores a task program as the last writer of each file it changed through fs.write or fs.write_json. A step that re-runs in consecutive builds for the same self-inflicted reason adds a RepeatedWarning.

Owners (“Cleaning up the cache”): every step record carries the tasks that own it, by TaskId. A run made with Engine::task_for adds its task to the owners of the record it stores, in the transaction that stores it, and keeps the owners of the record it replaces under the same key, a record marked as not cached included; a run without a task identity adds no owner and keeps the others. A step noted as cached, whether a hit on a stored record or a result reused within the build, makes its task an owner of the record under its key in the transaction of TaskSteps::finish that stores its position; a hit that refreshes a record’s observations keeps its owners. So after a build, every task that stores a step position for a key owns the record under that key. A path reset and kizz cache clean use the owners (see cleanup).

Shared state (“Kizz never panics”): the engine’s state that the threads of a build share is locked with kizz_threads::lock_within, never with Mutex::lock, which may panic when the calling thread holds the lock; a lock that stays held for the engine’s lock patience, kizz_threads::LOCK_PATIENCE unless Engine::with_lock_patience sets another, is an error of the step, or of the call that reads the state. The state of one task run lives in Cells that the private cells module changes, never in a RefCell, whose borrows panic while the cell is borrowed.

Structs§

Engine
What every task of a build shares to run, check, and record its steps.
NoStepEvents
Follows no step.
StepConnections
A run of a step that connected to IPv4 or IPv6 addresses or to Unix sockets, as the engine keeps it for the build’s warnings.
TaskSteps
The step executor of one run of a task program.
UnknownCalls
A run of a step that made system calls that the tracer does not know, as the engine keeps it for the build’s warnings.

Traits§

StepEvents
Follows the steps of a task run as they start or come from the cache, for the live output of kizz run.