Symptom

On a multi-server swarm, work posted on server A and won by an LLM on server B never reached DONE. It sat in RUNNING until the 60s lease expired, then requeued and re-auctioned forever. B’s inference completed and the result was persisted on B’s own LLM node — the failure was specific to getting that result back to the work node on A. Logs on A showed every cross-server wake for the work node being dropped:

1
2

A: dropping cross-server wake   x2887
A: cross-server wake for ...    x0

The same root cause also explained a second symptom: with the wake path dead, claims only landed via the periodic sweep fallback (~40s to ASSIGNED) instead of the ~5s the event path should give. A same-server control (poster and winner both on the same host) completed normally.

Root cause

PeerObservationClient.resolveSource(hostId, identity) resolves the node behind a SOURCE_TRIGGERED wake. hostId is the peer whose SSE stream the wake arrived on; identity is a self-describing NodeIdentity(nodeId, hostId) naming the node’s actual owner. For a cross-host swarm result these are different hosts: B publishes the wake (so it arrives over B’s stream), but the wake’s triggeringSource identity carries A’s hostId, because the work node lives on A.

resolveSource ignored identity.hostId and looked the node up on the SSE-source peer instead:

1
2

val peer = nodeManager.readNode(hostId) ?: return null
val fetched = runCatching { nodeHttp.readNode(peer, identity.nodeId) }.getOrNull() ?: return null

So A asked B (nodeManager.readNode(hostId) resolves B’s peer record, then nodeHttp.readNode fetches from B) for a node that only exists on A. The HTTP call 404’d, the fetch returned null, and the wake was dropped — every time, for every cross-host swarm job.

Fix

resolveSource now branches on identity.hostId, not the SSE-source hostId: if identity.hostId is our own install id, the node is read locally (nodeManager.readNode(identity.nodeId)); otherwise the peer to fetch from is resolved via identity.hostId, the node’s actual owner. The hostId SSE-source parameter is kept only for logging.

installId() reads a real file (/etc/krill/install_id or ~/.krill/install_id), so PeerObservationClient picked up the same seam already used elsewhere in the swarm code (ServerSwarmWorkProcessor, BeaconSettingsTask): a selfId: () -> String = installId constructor parameter, defaulted to the real platform value in production and injectable in tests. server/src/jvmTest/kotlin/krill/zone/server/events/PeerObservationClientTest.kt gained a case that publishes a SOURCE_TRIGGERED over a peer’s SSE stream for a node whose identity.hostId is the local host, and asserts the node is read locally and the peer’s /node endpoint is never called.

Prevention