Expand description
The kizz tracer for Linux.
LinuxTracer implements the tracer interface of kizz_trace with ptrace and
a seccomp-bpf filter that stops a traced process only for the system calls that
touch the file system, TRACKED_SYSTEM_CALLS, those that make the trace
incomplete, UNTRACEABLE_SYSTEM_CALLS, and the native calls outside the table
that it knows, KNOWN_SYSTEM_CALLS, and lets every other call run at full
speed. Only the kizz binary wires it in.
The tracer traces only the processes it starts itself: the child requests
tracing with PTRACE_TRACEME before it runs the program, and the tracer follows
the whole tree of processes and threads it starts, with the option that kills
all of them if the tracer exits. Nothing in the traced program changes, except
that io_uring_setup fails with ENOSYS, so its file accesses stay observable.
At each tracked call the tracer reads the call’s path arguments, directory
descriptors and flags from the stopped thread before the call runs, and its
result after it returns, and reports every path the tree touched as an absolute
path with its combined access kinds. Successful and failed calls are recorded
alike: a failed call, including a lookup that found nothing, is a probe. An
openat2 with RESOLVE_IN_ROOT or RESOLVE_BENEATH and without O_NONBLOCK
that fails with EAGAIN records nothing, though: Linux gives that error when a
rename or mount anywhere on the machine races with a .. step, and abandons the
lookup, so the caller may repeat the call, and each repeated call is recorded
like any other.
A process that makes one of UNTRACEABLE_SYSTEM_CALLS, which can change what a
path means or use a path in a way that the tracer does not record, marks the run
as incompletely traced when it makes the call, whether or not the call succeeds,
with a reason that names the process and the call; clone, clone3 and
unshare count only with CLONE_NEWNS, and setns only when it can enter a
mount namespace. A native system call outside KNOWN_SYSTEM_CALLS leaves the
trace complete: the tracer reports its number in the result’s unknown system
calls. Either call then runs unchanged.
The socket calls of TRACKED_SYSTEM_CALLS name socket addresses: a Unix socket
path is recorded like any path, a write for bind and a probe for the others,
and connect, and sendto, sendmsg and sendmmsg with a destination, also
report it in the result’s connections: each IPv4 or IPv6 address and port
whatever the outcome of the call, and each Unix socket, by path or abstract name,
unless its path was not found. Other socket families are not recorded.
At the exit of a successful call that opens a regular file for reading or runs
one, the tracer also notes the file’s metadata as its first read, unless the
tree wrote that path earlier or the call created or truncated the file: through
the descriptor that an opening call returned, and through the path for the
program of an exec and each interpreter and loader that the kernel loaded.
When the request asks for the process tree, the tracer also keeps, apart from
the combined accesses, which stay exactly as without it, each process of the
tree with the programs it ran and how it ended, and attributes what each call
touched to the program that its thread’s process ran, each kind with the call’s
outcome: ok, the name of the error, or unknown for a call whose result it
never saw. A successful exec belongs to the program that it starts, whose
arguments the tracer reads at the call’s entry. Connections, unknown system
calls and every reason that the trace is incomplete belong to programs in the
same way, while the result keeps the first reason. A command that cannot start
gives an error whose tree holds its one process, without a status, with the
request’s program and what the attempt touched. Without the request, the tracer
does none of this extra work.
LinuxTracer also answers the platform queries of the tracer interface. It
reads the extended attributes of a path with listxattr and getxattr, or
llistxattr and lgetxattr without following a final symbolic link, and a file
system that keeps none (EOPNOTSUPP) has none. It reads the machine state that
a command it starts inherits from its own thread: the kernel with uname, every
resource limit with getrlimit, the CPUs with sched_getaffinity, the
MemTotal and MemAvailable lines of /proc/meminfo, and the file system of
the working directory with statfs, and its mount flags with statvfs. It
names an error number with Linux’s name for it, such as ENOENT, the kernel’s
names for its internal restart errors, such as ERESTARTSYS, and errno and the
number for any other.
When a request is cancelled, the tracer interrupts the tree: it sends SIGINT
once to every live process. It follows the tree as before for
CANCEL_GRACE_PERIOD, and returns as soon as the whole tree has exited; when
the grace period ends first, it kills whatever remains with SIGKILL. A
cancelled run may therefore end with an exit code that the program chose after
it handled the interrupt. The program of a build runs in a process group of its
own, with pipes or /dev/null as its standard streams, so an interrupt that a
terminal sends to the tracer’s process group never reaches it directly: it
receives the interrupt exactly once, from the tracer.
A request that keeps the terminal (RunRequest::terminal), as kizz execute
makes, connects nothing: the program uses the standard input, output and error
of the tracer’s process directly, and stays in that process’s process group and
session, so a terminal’s input and signals, Ctrl-C included, reach it as they
would without kizz. Nothing is captured or echoed, no thread feeds or reads its
streams, and the result’s output is empty, in the shape of the request’s capture
mode. In both ways the program starts with an empty signal mask and SIGPIPE
at its default action, and inherits every other disposition, so a signal that
the tracer’s process ignores stays ignored in it.
The caller forwards a signal with Cancellation::forward; until the request
is cancelled, the watchdog delivers each one once, to the first process until
the tracer has collected its end, and afterwards to each live process of the
tree, signaling only the process IDs of the live set. The tracer knows the
first process by its identity, not by its process ID, which Linux may give to a
later process of the tree once the first is collected. A signal sent to the
whole process group of a program that keeps the terminal reaches it directly,
and once more when the caller forwards it, since neither can tell it from one
sent to the caller alone.
The tracer resumes a process that a stop signal stopped, because with
PTRACE_TRACEME it cannot learn when SIGCONT would end the stop. A terminal’s
Ctrl-Z therefore stops the tracer’s process, which keeps the default action of
SIGTSTP, SIGTTIN and SIGTTOU, and each traced process waits at its next
stop for the tracer, while a stop signal sent to one traced process alone does
not keep it stopped.
Each run starts its threads through kizz_threads, which waits and retries while
the system cannot start a thread for the moment, and never panics: the thread
that feeds stdin, kizz-feeder, and a thread that captures each output
stream, kizz-stdout and kizz-stderr, or kizz-output for merged output,
which start first, before anything of the request runs; then the tracing
thread, kizz-tracer; and the watchdog of cancellation, kizz-watchdog. When
one still cannot start, no process of the request is left, and the run returns
TraceError::Tracer, whose message names the thread and the system’s error.
The cleanup of last resort is the end of the tracing thread: every process and
thread of the tree is its tracee, with PTRACE_O_EXITKILL, so when the thread
ends, for an error, a panic or the end of the tree, Linux kills whatever is left
of the tree, also processes that left its process group or session and threads
in stops that the tracer already collected. The run then collects its first
process through a process descriptor, and its output and input threads end with
the pipes. No step of that reads /proc or signals a process or thread ID.
Collecting by descriptor needs Linux 5.4 or later: the descriptor comes from
pidfd_open, and waitid with P_PIDFD collects the process.
Structs§
- Linux
Tracer - Runs commands under
ptracewith a seccomp-bpf filter in front. - Tracked
System Call - A system call that the filter stops for, with its
x86_64name and number.
Constants§
- CANCEL_
GRACE_ PERIOD - How long a cancelled tree may run after its interrupt, the
SIGINTthat every live process receives once, before the tracer kills whatever remains of it. - KNOWN_
SYSTEM_ CALLS - The native system calls that the tracer knows, as ranges of numbers: those of
the
x86_64table of Linux 7.3,arch/x86/entry/syscalls/syscall_64.tbl, which uses 0 through 336 and 424 through 472 for native calls. Any other native number is a system call that the tracer does not know: 337 through 423, which the table leaves unused, and every number above 472. - TRACKED_
SYSTEM_ CALLS - Every system call the filter stops for, in the order of the spec’s list: opening files, metadata and existence, file systems, watches, extended attributes, links, directory listings, programs, the working directory, changes, changes through a descriptor, and sockets.
- UNTRACEABLE_
SYSTEM_ CALLS - Every system call that marks a run as incompletely traced, in the order of the
spec’s list: opening by handle, changing the root, mounting, mount namespaces,
and the other uses of a path. The first four groups can change what a path
means; the last uses a path in ways that the tracer does not record.
unshare,cloneandclone3mark the run only withCLONE_NEWNS, andsetnsonly when it can enter a mount namespace.