What happened

In the “Local LLM Nodes” demo, an LLM node (Sentry) watching a camera frame produced the right answer and then overwrote it with the wrong one:

1
2
3
23:07:13 Verdict  FLAG: A large unfamiliar dog standing at the run fence
23:07:13 Verdict  NORMAL          <-- stale inference lands second and wins
23:07:15 Gate     0.0             <-- downstream reads NORMAL; nothing wakes

Two inferences were in flight for the same node. One had resolved its inputs before the new camera frame propagated and returned NORMAL; the other saw the dog and returned FLAG. The snapshot write was last-write-wins, so whichever HTTP call finished last won — and on this run that was the one holding the stale input. The Lambda gate, the threshold, and the 32B model it guards all read NORMAL, and the alert never fired.

On other runs the ordering came out the other way and the chain worked. That is the part that matters: it was not broken, it was nondeterministic. The demo shipped with a workaround — the camera re-publishes the same frame a few seconds later so a fresh inference wins — which hid the defect rather than fixing it.

Root cause

ServerLLMProcessor.process() did scope.launchProcessing { runInference(node) } with no per-node coordination at all, so N invocations meant N concurrent inferences. Each one:

1. resolved its inputs (resolveInputNodes), then
2. spent up to five minutes inside callBackend, then
3. wrote its result with nodeManager.update(successNode).

Nothing between steps 1 and 3 checked whether the inputs it had resolved were still current, and nothing stopped a second invocation from starting while the first was parked in step 2. Worse, runInference wrote back a node.copy(...) derived from the Node captured at invocation time, so even the metadata it persisted could be stale.

The window is enormous — an inference is a multi-second-to-multi-minute HTTP call against a local model — so any node whose source updates faster than the model answers will hit this. The camera in the demo did exactly that.

Fix

• New ConflatingGate (server/src/jvmMain/kotlin/krill/zone/server/ConflatingGate.kt): serializes work per key and conflates the queue — at most one block running per key, at most one waiting, redundant invocations dropped.
• ServerLLMProcessor runs EXECUTE through the gate keyed by node id, and re-reads the node inside the lock (nodeManager.readNodeStateOrNull(node.id)) rather than trusting the copy it was invoked with.

Together those give the invariant that actually matters: inputs are resolved inside the lock, so a waiting inference always observes what the previous one wrote, and the last write is therefore always the one derived from the freshest inputs.

Conflation (rather than queueing every invocation) keeps a burst of source updates from building a backlog of inferences that are obsolete before they start. Dropping a redundant invocation is safe precisely because the run already waiting has not resolved anything yet — when it runs it reads exactly what its duplicates would have. It must not replace the waiting run, though: that would let a steady stream of updates starve the node forever. There is a test for that (a run that arrives after the queue drains is not dropped).

Prevention

• “Resolve, do slow thing, write” is a read-modify-write, and needs the same discipline. It is easy to see the race in a counter++, and easy to miss it when the three steps are spread across a coroutine and an HTTP call. Any processor that reads node state, awaits I/O, and then writes node state has this shape. ServerLambdaProcessor and ServerCameraProcessor should be audited for it.
• Last-write-wins is a choice, not a default. nodeManager.update() silently accepts whatever arrives last. Where the value derives from inputs read earlier, “last” and “freshest” are not the same thing, and the API does nothing to tell you which one you have.
• A workaround that makes a nondeterministic bug usually-work is a liability. The demo’s frame re-publish made the chain succeed often enough to ship, which is exactly why the underlying race survived to be filed later.
• When virtual time cannot reach the code, restructure the code — don’t fake the clock. The concurrency invariant could not be tested through ServerLLMProcessor because Ktor’s engine dispatches off the test scheduler, so advanceUntilIdle() returns while the call is still in flight. Rather than reaching for delay() and a wall-clock assertion (which CLAUDE.md bans, and which would have been a flake), the coordination logic was extracted into ConflatingGate — pure coroutines, fully owned by the test scheduler, six deterministic tests. The processor test then only has to prove the re-read, which needs no race window at all: hand it a deliberately stale node and assert the request carries the current one. A concurrency invariant you cannot test deterministically is usually one that is tangled with I/O it does not need to be tangled with.