How verification runs work
When someone claims a bounty and sends back a patch, somebody has to run it. That means executing code a stranger wrote. This page explains where that code runs, what it can reach while it runs, and who can read the result today.
No prior knowledge assumed. If you only read one thing, read the first diagram.
The short version
CONTRIBUTOR the result prints
writes a patch ──────────► a run happens ──► on their machine
│ (it goes nowhere
▼ else — see
"Who reads it")
┌───────────────────────┐
│ a locked-down │
│ container: │
│ • install phase │ ← network, but only to an
│ • network CUT │ allowed list of hosts
│ • test phase │ ← NO network at all
└───────────────────────┘
Two phases, and a cut between them. That cut is the whole idea. Dependencies get installed with a narrow door to the internet; then the door is removed and the tests run with no network interface at all.
Why the cut matters
A test suite with a live internet connection can pass for reasons that have nothing to do with the code. It can download a fixture, phone home, or quietly fetch a dependency that was supposed to be installed already. It can also send your source somewhere.
Cutting the network between install and test removes that whole category of question. If the tests pass with no network, they passed on the code.
PHASE 1 — install PHASE 2 — test
┌────────────────────┐ ┌────────────────────┐
│ container │ │ container │
│ │ │ │
│ npm install │ │ npm test │
│ │ │ │ │
│ ▼ │ │ (no interface, │
│ ┌───────────┐ │ │ not a blocked │
│ │ egress │ │ │ one — there is │
│ │ proxy │ │ │ nothing to │
│ └─────┬─────┘ │ │ connect to) │
│ │ │ │ │
└────────┼───────────┘ └────────────────────┘
▼
only allowed hosts
(package registries)
The proxy runs outside the container, in a process the container cannot reconfigure or bypass. Reaching it is the point — during install it is the only route out, on an internal Docker network with no gateway, and the container finds it by name. What the container cannot do is edit the allowed list, talk past the proxy, or open a connection that does not go through it. So the list is enforced, not advisory: the thing deciding is not inside the box.
What the container is allowed to do
The container is deliberately boring. Every one of these is a default we turned off rather than a feature we added.
| Setting | Effect |
|---|---|
--network=none (test phase) |
No network interface exists |
--cap-drop=ALL |
No Linux capabilities — cannot raise its own privileges |
--security-opt=no-new-privileges |
Cannot gain privileges by running a setuid binary |
--pids-limit=512 |
Cannot fork-bomb the host |
--rm + --init |
When the work exits, the container is gone |
| runs as a normal user | Not root — on macOS and Linux; see the note below |
On Windows the container runs as root. The --user flag is built from the host's
own uid and gid, and Windows has neither, so it is omitted and the container keeps the
image's uid 0. Root inside a container with no capabilities, no new privileges and no
network is a much smaller thing than root on a machine — but it is not the same as the
line above, and you should know which one you are getting.
It sees three directories and nothing else of your machine:
/fenced/clone the checked-out repository (read + write)
/fenced/jail a writable HOME, holding the (read + write)
package caches for node, go
and rust
/fenced/tmp scratch space, a real tmpfs (read + write)
Install and test are two separate containers, each started with --rm, so what carries
between them is only what lives outside the container. That is the jail — and the clone:
both are folders on your disk, and the jail stays one precisely so a package cache
survives the install step.
Which means the caches behave differently by ecosystem. Node, Go and Rust have their
cache variables pointed into the jail, so their downloads survive install and are still
there for the tests. Python and Ruby have no such variables, so their downloads land
inside the container and go when it does. Scratch written to /fenced/tmp does not carry
over either; it is memory, and the second container starts with an empty one.
None of it survives the run itself. The whole scratch directory, jail included, is deleted when the command exits.
/fenced/tmp is a memory filesystem rather than a folder on your disk. That is
not a detail — a bind-mounted folder does not enforce file permissions on some
Docker setups, which once made a project's own permission test fail and got
blamed on the contributor rather than on us.
Where the run happens
Today, a run happens on the contributor's own machine. They run it, and they read the result — it does not travel anywhere on its own.
That is genuinely useful — it catches broken patches before anyone else spends time on them — but be precise about what it proves. It proves the tests passed in a locked-down container. It does not prove whose container.
TODAY
┌──────────────────────────────┐
│ contributor's machine │
│ their Docker │ ──── result ────► printed for them,
│ the fence above │ and stays there
└──────────────────────────────┘
WHAT'S COMING (not available yet)
┌──────────────────────────────┐
│ a machine neither party │
│ controls │ ──── result ────► founder reads it
│ same fence, same steps │
└──────────────────────────────┘
The design keeps where as the only variable. The steps, the container settings, and the cut are identical wherever the run happens — so moving the run does not quietly change what the run means.
Status, plainly
| Tier | Where it runs | Available |
|---|---|---|
| Contributor's Docker | the contributor's own machine | Yes, today |
| Hosted pool | a machine we boot per run | Not yet |
| Confidential VM | hardware-isolated machine | Not yet |
The hosted and confidential tiers are being built. Until the page says otherwise, assume every run you see came from the contributor's machine. We would rather say that plainly than let you infer something stronger.
Who reads it
Today: the person who ran it, and nobody else. The run prints its result to the terminal it was started from. Nothing uploads it and no founder is notified.
The run also opens a preview holding the full run document, but that preview is bound to loopback — it is reachable from the machine that ran it and from nowhere else, and the code refuses to bind it any wider without a credential no command-line path supplies. Sending someone the link does not let them open it; they reach their own machine. Treat it as a better view for the person who ran the run, not a way to show it to anyone.
The founder-side reader exists — a founder can pull verification evidence for their own bounty — but the half that would send a run to it has not been built, so there is nothing sitting there to read. If you are paying for a bounty, assume you have seen no run unless a contributor pasted you the output.
We would rather say that than describe a delivery that does not happen.
What the run reports, wherever it is eventually read:
- whether it judged the work at all — a run either produced a verdict or refused before starting, and a refusal is reported as a refusal
- what happened to the suite, as one of a named set of answers — the tests passed, the tests failed, no tests were observed, the counts could not be parsed, the test command was unavailable, the environment gave out, the budget ran out
- the test command, where it came from, and its exit code
- a tail of the test output
- whether anything was left running afterwards
Those first two lines matter more than they look. Only one answer in that set means the patch might be at fault: the tests failed. A refusal means the run declined to judge at all — a missing input, a patch it could not bind to the code it tested. An exhausted environment or an expired budget means our side gave out. Reading any non-green result as "the patch is broken" blames a contributor for something the patch never did, which is why these are separate answers instead of one bad result.
The last one is a real check, not a formality. The run counts what it started and counts again after teardown, and a mismatch is reported rather than ignored.
Things this does not do
Worth stating, because the gaps matter more than the features.
- It does not grade the contributor. A run says what happened to a patch. It is not a score, a screen, or a badge, and no part of the product gates anyone on it.
- It does not guarantee a patch is correct. Passing tests mean the tests passed. A maintainer still reads the diff.
- It does not run your production code against production data. It runs a checked-out repository in a throwaway container.