Push scans are narrowed by a claim rule that fails open

Decision record 0010

Amended by 0051: an ignore entry may be a mapping with a glob and a reason, and a stack it leaves out is listed with the reason in a fold under In sync. It still has no row.

The first real user has 58 stacks, a pool of at most four self-hosted runners with 1 CPU and 4 Gi each, and about nine commits a day that arrive in bursts. Two out of three of those commits change files in one project directory only. Previewing everything on every push would keep that pool busy for most of an active day, so a scan that follows a push is a narrowed scan: it previews only the stacks that claim a changed file. A scan on schedule, on manual dispatch or from the rescan box is always a full scan.

A stack claims a changed file when the file lies inside the stack’s directory, or matches one of that stack’s inputs globs. A changed file that no stack claims makes the scan a full scan. That one rule is what keeps narrowing honest in a monorepo: a shared components package, a root lockfile, a shared tsconfig or an env file all sit outside every stack directory, so changing them previews everything, without Sluiceway knowing what they are.

Working out each stack’s real inputs from workspace dependencies or an import graph was rejected. It needs per-language knowledge in the adapter, and it still misses what programs read at run time. In the homelab every one of the 51 projects imports the shared package, environment variables outrank Pulumi config, and one stack reads the dashboards folder of every app. Inverting the rule, so that an unclaimed file is ignored unless it is on a configured shared list, was rejected because zero config would then be silently wrong for any repo with a shared package.

Consequences

The changed files are the comparison from the scan-sha on the dashboard’s root marker (0009) to the commit the scan checked out. The push event’s own before and after are not used: only the newest waiting scan survives (0004), so the scan for an earlier push may never run, and its files must still be covered by the next one.
The list comes from GitHub’s compare API, not from local git, because the default checkout is one commit deep and core/ does not shell out to git. The glue hands core a list of paths. A renamed file counts under both its old and its new path.
The scan is a full scan whenever the comparison cannot be trusted: there is no dashboard yet, the root marker is missing, unreadable or of another version, the comparison is not a straight line from scan-sha to the checked-out commit (a force push, or a re-run of an old workflow run), the file list is at the API’s cap of 300 files, or the call fails.
Both stacks of a project that has prod and dev stack files claim a change in their shared directory. When one stack directory contains another, a file in the inner one is claimed by both.
inputs is an optional list of globs on a stack’s config entry, relative to the repo root. It adds claims and never removes them. It covers a stack that reads another directory, and later a shared package that only some stacks use. The homelab’s workspaces/pi stack lists workspaces/apps/*/dashboards/**.
scan.unrelated is an optional top level list of globs for files that claim nothing and force nothing, such as **/*.md. It is empty by default. No defaults were chosen because a program can read any file, and a wasted full scan costs far less than a row that is stale on a fresh install.
sluiceway.yaml and the workflow files lie outside every stack directory, so changing them gives a full scan with no special case.
A narrowed scan also previews two kinds of stack that claim nothing: a discovered stack that has no row on the live dashboard, and a stack whose row is a preview failure. The second reads the row state, which is safe because previewing too much is never wrong (0009 keeps row state out of every deploy decision). A transient failure then heals on the next push. A stack that stays broken costs one extra preview per push.
The ignore globs from the brief match the stack id (0006), not the path alone. Path patterns keep working, and **/*:dev drops the second stack of a directory, which a path cannot express. An ignored stack has no row, is never previewed and claims nothing.
The rule can miss. A stack whose program reads an undeclared file inside another stack’s directory, or an input that is not a file at all (an output of another stack, a remote chart, a secret’s value), shows a stale row until the next full scan. No deploy is ever wrong because of it, since apply previews again and compares the hash before it deploys (0008). This is why the scheduled full scan is part of the design and not an optional extra, and why the docs must tell users to keep it.
Attribution of pending changes to merges is decided separately. It should use this same rule for “which commits touched this stack”, so that a row a narrowed scan carries through never lacks a commit that a full scan would have listed.

Settled while building (slice 1.12)

Only the event name push gives a narrowed scan. Every other event is a full scan and makes neither the first read nor the compare call, so it costs the requests it cost before.
The compare call is GET /repos/{owner}/{repo}/compare/{scan-sha}...{sha} with per_page=1. Checked against the real API: the page size only counts commits, and page one holds every file of the whole comparison up to the cap of 300, whatever the page size. One request, and no commit is read.
A straight line is GitHub’s status ahead. identical counts as one too: it is the same commit scanned again, a re-run of the newest run, and nothing changed. behind, diverged and any word GitHub adds later are a full scan. A scan-sha that a force push took away makes the call fail with 404, which is a full scan as well.
A scan-sha that is not a full commit id (40 or 64 hex characters) is unreadable. No request is made with a value a person typed into the marker.
A list of 300 files or more is at the cap. A list of 299 is trusted.
scan.unrelated is applied first. A file it matches claims nothing, also when it lies inside a stack’s directory or matches an inputs glob. That is what makes **/*.md useful.
A stack at the repo root has the path . and claims every file. In such a repo no file is ever unclaimed, so a push is never a full scan because of the claim rule.
An ignored stack claims nothing, so a changed file in its directory has no claimant and gives a full scan, unless scan.unrelated lists it.
The job log says which kind of scan ran and why. A full scan: This is a full scan: the event is schedule, and only a push gives a narrowed scan. A fall back: This is a full scan. A push gives a narrowed scan, and this one fell back to a full scan: no stack claims package.json and 2 more changed files. Every unclaimed file is then listed in a group of its own, with a line that points at inputs and scan.unrelated. A narrowed scan says how many files changed between which commits, and one line per previewed stack: it claims a file, it has no row, or its row is a preview failure.

Settled while building (slice 2.13)

A change to sluiceway.yaml at the repo root still falls back to a full scan by the claim rule, with no special case in the plan. Only the words change, because “no stack claims sluiceway.yaml” read as if a stack should (onboarding log, hurdle 14): This is a full scan. A push gives a narrowed scan, and this one fell back to a full scan: sluiceway.yaml changed, so every stack is previewed. Other unclaimed files in the same push follow as , and no stack claims package.json and 1 more changed file, and only those are listed in the group that points at inputs and scan.unrelated. A file of that name in a subdirectory is an ordinary file.