Policies run against the preview, and a hard failure takes the box off the row

Decision record 0106

Amends 0013 (one more tool the workflow installs and Sluiceway starts, never wraps), 0021 and 0022 (the tool’s own preview document is written to a file for the policy runner and removed, and a policy’s message is the one text from outside Sluiceway’s code that reaches the dashboard, as the repo’s own words, escaped), 0027 (the lines of a pending row), 0050 (the preview page shows the policies as the row does), 0083 (a bulk box skips a row a policy stopped), 0095 (a failed policy stops a deploy on merge outright) and 0009 (one more row marker key). Built as slice 5.41.

Every tool Sluiceway is compared with gates on policy: Atlantis with Conftest, Spacelift with Rego, Terrateam with OPA, Conftest and Checkov, HCP with Sentinel and OPA (issue 229). Sluiceway is well placed for it: the fresh preview per stack is the JSON those runners want, and the tick is already a gate, which none of them has. The owner decided on 2026-09-24: Conftest is the runner, installed by the workflow and run by Sluiceway against the preview, never wrapped; a policy reads the preview only, because that is what a tick approves; hard failures only in this slice, a broken policy takes the box off the row and names the policy in its own words, escaped; a policy that fails to run is a warning line, not a failed policy; any failed policy stops a deploy on merge outright; the soft failure that asks for a second ticker is deferred with the door open.

Decision

policies in sluiceway.yaml names directories or files of Rego policies, relative to the repo root and inside it, like a stack’s path, and stacks[].policies adds paths for the stacks of an entry, the way inputs add up. Empty, and nothing runs: a repo without the key is byte for byte what it was, and a stack with no policies is never asked for its document. The check does not list the paths yet (docs/later.md).
Conftest is the runner, and the workflow installs it, as it installs the tool (0013). Sluiceway starts conftest test --output json --all-namespaces --no-color --policy <path>... <document> in the repo root, with the environment the stack’s tool gets, minus INPUT_*, and the stack’s own time limit. Every namespace of every file in the paths runs, so a rule in a package the person did not name main is not passed over in silence. The minimum version is 0.50.0, and the fixtures were recorded with it and with 0.70.1: the report, the flags and the exit codes are the same on both. conftest --version is asked once per job, before the first policy runs.
A policy reads the tool’s own preview document of the stack, and nothing else: Pulumi’s preview JSON as pulumi preview --json printed it, the plan JSON of OpenTofu and Terraform as show -json printed it, and for a Helm release and a directory of Kubernetes manifests the manifests the preview renders, as a YAML stream that Conftest judges object by object. That is what a tick approves, and it is what the policies people already have are written against. The program’s files are not an input (docs/later.md). An adapter hands the document back only when asked (keepDocument on PreviewOptions), so every preview that does not ask reads as it always did, and the Helm adapter renders the manifests once more for it, as it does for a saved plan.
The document holds values (0021), so it lives on disk only while conftest runs. The runner writes it to a file in a directory of its own, readable by nobody else, starts conftest, and removes the directory on every way out. The scan drops the document from the preview result as soon as the policies ran. Nothing else takes it: not a row, the summary, a page, the result file, an annotation or the job log. Conftest’s stdout is the report, which the core reads and nothing prints; its stderr goes to the stack’s group of the job log with the tool’s other words (0022).
Only a pending stack is tested: one whose preview gave changes. A stack in sync has nothing a deploy would do, a drifted row with nothing from the code is a repair of reality and no change of the code, and a preview that failed has no document. So the cost is one conftest run per pending stack per scan, in the stack’s slot of the pool, right after its preview and its drift check.
A failed policy is a hard failure: the row has no box until it passes. The first line of the row has no box, the marker carries policy="failed", and the lines under the attribution line read :no_entry: **2 policies failed**, so this change has no box until it passes: and then one line per failure, :no_entry: <code>main</code> · <the message>, at most five, the rest counted with a link to the preview page. A redacted or shortened row keeps the count and the link and names no policy. The page names every failure whole, first, before the changes, and so does the summary. The next scan that finds the change passing draws the box again; nothing is remembered between scans.
The message is the policy’s own words, escaped as untrusted text. It is the one text from outside Sluiceway’s code that reaches the dashboard, and it is the repo’s own: a .rego file on the default branch, reviewed like dashboard.showValues is (0052). Every character Markdown or HTML acts on is escaped by the same function that escapes a resource name (0027), a control character becomes a space, so a message cannot start a line, a row or a marker, and a row cuts a message at 200 code points. A policy can print a value into its message, as showValues can show one, and the docs say in one line to name what is wrong, not what it is set to. The namespace is escaped the same way. The message is display text and nothing is decided from it.
A policy that could not run is a warning line, not a failed policy. conftest missing from the runner or older than 0.50.0, a policy that does not parse, a policy path the checkout does not hold, a report that could not be read, a run out of time: the row keeps its box and carries :warning: the policies did not run: <reason>. Nothing was checked, see the run., the reason a constant of Sluiceway’s own (0022), the run carries a warning annotation, and the job log holds conftest’s words. conftest missing or too old is one warning for the job that says what to install, and every pending row of a stack with policies gets the line. The job never goes red for it: a policy that fails to run is not a policy that failed, and a red job on every push for a runner without conftest would teach people to ignore red (0012). The check does not warn about a workflow without a conftest step yet (docs/later.md).
A warn rule changes nothing on the row. It is listed on the preview page and in the summary, under the failures, and in the job log. It is not the soft failure of the issue: that one asks for a second person, and is deferred.
Any failed policy stops a deploy on merge outright. A stack set to deploy: on-merge whose change fails a policy is not handed to the on-merge decision at all: it opens no record, its row carries the policy lines and no on-merge note, and the job log says <stack> deploys on merge, and this change does not: 2 policies failed. It counts as a change nobody ticked, so a stack that depends on it waits, as it waits on any change waiting for a tick (0056, 0095).
A tick on such a row is refused by the marker, not by the box. The row has no box, so a tick can only come from a hand-edited body. resolve reads policy="failed" from the live row and clears the tick with a note, nobody is looked up and nobody is mentioned, in the order of the other refusals: a taken stack first, then deploys: false, then the policy. The bulk box counts no such row and the confirm box names none (0083), so a bulk tick cannot deploy it either. Without this, write access to the issue would be enough to get around a policy that only a merge to the default branch can change.
The preview page and the summary show what the row shows (0050): the page’s summary says 2 policies failed, so the row has no box until the change or the policies change, every policy passed with the count of rules and the namespaces, or that the policies did not run and why; its text lists every failure and warning first, whole, and the changes give way to them when the page is cut. The summary lists the same under the stack’s counts, at every level.

Considered

Sluiceway’s own diff as the input, the canonical document the hash covers (0008): one shape for every tool, no value in it, and exactly what a tick approves. Rejected: a policy about infrastructure is about values, an instance type, a public bucket, a replica count, and the policies people have are written against the tool’s plan. The issue said the preview is exactly the JSON Conftest wants, and the owner decided for the preview.
The program’s files as an input too. Left out, as the owner decided: a tick approves the preview, not the source. Conftest can read a directory of files, so a later slice can add it without a breaking change (docs/later.md).
A failed run of conftest as a hard failure, so that a runner without conftest cannot pass a change a policy would have stopped. Rejected by the owner: a policy that fails to run is not a policy that failed. The cost is stated: a workflow that never installs conftest gets a warning on every scan and keeps its boxes. The docs say so, and the check will say it (docs/later.md).
Only the main namespace, as Conftest does by default. Rejected: a policy in another package would run nowhere and say nothing, and every namespace can still be narrowed by what a person puts in the paths.
Naming the failures in a comment, or the failed policy on the counts line or in the header. Rejected: the row is where a person decides (0032), a comment is a notification in every inbox, and a policy failure is a fact about a change like a destroy is.
Checkov as a second runner. Deferred with the owner’s decision; the document and the runner seam are made so it can follow.
Running the policies again in apply, before the deploy. Left out: the row a person ticks has a box only because the policies passed on the scan of that commit, apply holds the deploy to the hash of that same preview (0008), and the policy files are on the same commit. A later slice can add it if a policy ever reads something a commit does not fix.
The soft failure that asks for a second ticker. Deferred by the owner: it needs a new row state, two tickers walked from the edit history and a record carrying both, and that is where a bug about who may deploy would live. Record 0020 rejected a built-in second approval as a general setting; a second tick that a policy asks for is the one shape the owner keeps the door open for, and it is built when a team asks for two signatures.

Consequences

core/policy.ts holds the command line, the version rule, the reading of the report and the decision, so every mode reads a run the same way and a hosted version can reuse it. src/policy/conftest.ts is the runner: the temporary file, the process through the runner seam, the check of the version. Both stay behind the boundary rule with core/, adapters/ and render/.
PreviewOptions gains keepDocument and a successful PreviewResult gains document, { text, format }, which nothing but the runner may take. Four adapters hand it back.
PendingRow gains policies, the row marker gains policy="failed", documented in docs/what-sluiceway-writes.md, and ParsedRow reads it. A row a policy stopped is placed and counted as pending: the header, the counts line and the crate count do not change.
The scan checks conftest once, tests each pending stack with policies in its pool slot, drops the document, and hands the outcome to the row, the page, the summary, the on-merge decision and the job log. resolve refuses a tick on a stopped row with a note and a line in the job log.
The fixtures under test/fixtures/conftest/ are recorded by scripts/record-conftest-fixtures.ts from the policies and inputs in scripts/fixtures/conftest/, with the real binary, on 0.50.0 and 0.70.1. Conftest 0.70.1 stops without a HOME, which a job always has.
CONTEXT.md gains Policy and Preview document. docs/later.md loses the policy engine from the bigger efforts and gains the second ticker, the check’s lines, Checkov, the program’s files and the pull request preview. The security page says what a policy’s message is and who can read it.