On a 4-server simulated swarm (one GPU-less poster, three swarm-enabled
Server.LLM nodes), a single decomposable Swarm.Work job with
maxSubtasks: 3 structurally worked — children fanned out, ran in
parallel — but every step that should cost ~10s (a fixed 6s inference stub
plus a claim window) cost 55-100s, and the job usually never reached DONE
inside 5 minutes. It was not lease expiry: the work host logged requeue 0
times. The work host’s logs showed the real story:
1
2
cross-server wake for … 92,369
dropping cross-server wake 33
~300 successful cross-server wakes/second for one job on a 3-LLM-node
fleet. Tightening claimWindowMs/leaseTtlMs made it worse (a lease now
expired mid-flight before the backlog cleared).
SwarmWatchTask.sweep() gives every swarm-participating server a blanket
SSE subscription to every peer Server node’s /events stream, so it
can catch OPEN Swarm.Work posted anywhere in the fleet. On a small swarm
this makes every server subscribe to every other server — a full mesh.
PeerObservationClient.handleEvent’s SOURCE_TRIGGERED branch mirrors the
triggering node and unconditionally republishes it via
ServerNodeManager.publish, which re-broadcasts a fresh SOURCE_TRIGGERED
onto this host’s own /events stream whenever the hop-TTL budget (design
D11, default 16) is still positive — by design, so a legitimate multi-hop
chain (A wakes B, C only observes B) still reaches C. That decrement is
correct for a single path, but nothing tracked whether a given wake had
already been relayed. In a full mesh every host is directly reachable from
every other host, so the “further hop” the TTL budget exists to support is
always redundant: the same (triggeringSource, epoch) wake arrives at each
host multiple times (once per peer that also relays it), and each arrival
is relayed again to the whole mesh. Total traffic for one genuine state
change grew with the mesh’s branching factor raised to the hop-TTL power
instead of linearly with host count, saturating the event pipeline that
claim windows, lease heartbeats, and result collection all share — which is
what actually produced the 20-50x latency inflation, not anything wrong
with the claim/lease timers themselves.
PeerObservationClient now remembers wakes it has already relayed, keyed
by "${payload.triggeringSource}:${payload.epoch}". Both fields are
preserved verbatim across every hop of a relay chain (the identity names
the node’s actual owner; the epoch is the root propagation’s monotonic
counter), so the pair uniquely names one propagation regardless of which
peer relayed it. A duplicate arrival is dropped before it can re-announce,
bounding total traffic for one wake to O(hosts) instead of
O(branching^hops). The hop-TTL mechanism itself is untouched — it still
correctly bounds genuine linear multi-hop chains.
server/src/jvmTest/kotlin/krill/zone/server/events/PeerObservationClientTest.kt
gained a case that delivers the same (triggeringSource, epoch)
SOURCE_TRIGGERED via two different peer hosts and asserts the mirrored
node is only republished once.
SwarmWatchTask can discover OPEN work anywhere) silently creates a full
mesh. When a subsystem does this, treat the relay path downstream of it
as topology-hostile by default — assume every message can arrive by more
than one route — rather than assuming the origin’s hop-TTL alone keeps
volume in check.