Decentralized Mesh Network Peer

Decentralized Mesh Network Peer: Resilient Architecture

The Krill Platform is designed as a resilient peer-to-peer network swarm of servers. Each server in the mesh network connects to other servers, known as Peers, to share data and resources. This decentralized architecture enhances reliability, scalability, and fault tolerance, ensuring that the network remains operational even if individual servers go offline.

Overview

Peers represent the connections between Krill servers on your local network. Unlike External Servers that require manual configuration, Peers are automatically discovered through beacon broadcasting. This creates a self-organizing mesh where servers find and connect to each other without user intervention.

Key Features

• Auto-Discovery: Servers automatically discover peers on local network via UDP beacons
• Cluster PIN Trust: All servers sharing the same 4-digit cluster PIN automatically trust each other – no manual key exchange required
• Beacon Broadcasting: UDP beacon system for server announcement, including a rolling TOTP cluster token for membership validation
• Decentralized Architecture: No single point of failure
• Data Synchronization: Share node states across the mesh
• Workload Distribution: Distribute processing across servers
• Fault Tolerance: Network continues if servers go offline
• Reconnection: Peers re-establish SSE connections when servers return (fully automatic mesh self-healing is on the roadmap)
• Real-Time Updates: SSE (Server-Sent Events) based state synchronization

Mesh Network Architecture

graph TD
    subgraph "Local Network Mesh"
        A[Krill Server A] <-->|Peer Connection| B[Krill Server B]
        B <-->|Peer Connection| C[Krill Server C]
        A <-->|Peer Connection| C
        A <-->|Peer Connection| D[Krill Server D]
        C <-->|Peer Connection| D
    end

    E[Client App] -->|Connect| A
    E -->|Connect| B
    E -->|Connect| C
    E -->|Connect| D

    F[Beacon UDP] -.->|Discovery| A
    F -.->|Discovery| B
    F -.->|Discovery| C
    F -.->|Discovery| D

How Peer Discovery Works

The peer discovery and trust process is fully automated when servers share a cluster PIN:

1. Beacon Broadcast: Each server broadcasts UDP beacons on the local network (multicast group 239.255.0.69, port 45317, TTL=1). Beacons include a rolling TOTP cluster token derived from the shared PIN.
2. Beacon Validation: Receiving servers validate the cluster token in the beacon to confirm the sender belongs to the same cluster.
3. Certificate Download: Server downloads the peer’s self-signed certificate via GET /trust (TOFU model, only after beacon validation confirms cluster membership).
4. Health Check: Server verifies connectivity with GET /health using the shared PIN-derived Bearer token.
5. SSE Connection: Servers establish SSE (Server-Sent Events) streams using the shared PIN-derived Bearer token for real-time node state synchronization.
6. Auto-Reconnect: Lost connections are automatically re-established on the next beacon cycle.

All servers in a cluster derive the same Bearer token from the same PIN via PBKDF2. No key exchange or handshake is needed – cluster membership is validated by the beacon’s rolling TOTP token, and ongoing authentication uses the shared derived Bearer token.

Peer Communication Flow

sequenceDiagram
    participant A as Server A
    participant B as Server B

    A->>B: UDP Beacon (with rolling TOTP cluster token)
    B->>A: UDP Beacon (with rolling TOTP cluster token)
    Note over A,B: Both validate cluster tokens
    A->>B: GET /trust (download certificate)
    B-->>A: Self-signed certificate
    A->>B: GET /health (Bearer: PIN-derived token)
    B-->>A: 200 OK
    Note over A,B: Both servers share the same PIN-derived Bearer token
    A->>B: SSE /sse (Bearer: PIN-derived token)
    B->>A: SSE /sse (Bearer: PIN-derived token)
    Note over A,B: Real-time node state sync via SSE

Peer Metadata

Each peer connection maintains metadata about the remote server:

Field	Description
name	Server hostname
port	Server communication port
model	Hardware model (Raspberry Pi, etc.)
version	Krill Server version
os	Operating system
platform	Platform type (LINUX, WINDOWS, MAC)

Cluster PIN

The cluster PIN is a 4-digit numeric code shared across all servers and clients in a Krill cluster. It is the single source of trust for the entire mesh.

Credential Files: /etc/krill/credentials/pin_hash and /etc/krill/credentials/pin_derived_key

Property	Value
Format	4-digit numeric code
Derived Token	PBKDF2 with cluster-specific salt
Beacon Token	Rolling TOTP derived from PIN
Permissions	0400 (read-only)
Set During	apt install krill (installer prompt) or app FTUE

Setup for multiple servers:

# During installation of each server, enter the same 4-digit PIN
# when prompted by the Debian installer.

# To reset the PIN on an existing server:
sudo krill-reset-pin

Use Cases

• Home Automation: Multiple Raspberry Pi servers throughout the house
• Lab Monitoring: Distributed sensors across a facility
• Redundancy: Multiple servers for critical automation
• Scalability: Add servers as your needs grow
• Load Distribution: Spread data collection across devices
• Geographic Distribution: Servers in different rooms/areas
• Edge Computing: Process data close to sensors

Benefits of Mesh Architecture

Reliability:

• No single point of failure
• Network continues if servers go offline
• Automatic reconnection when servers return

Scalability:

• Add servers without reconfiguration
• Linear scaling with additional nodes
• No central bottleneck

Performance:

• Local processing reduces latency
• Distributed workload
• Efficient resource utilization

Simplicity:

• Enter the same 4-digit PIN on each server during install
• Auto-discovery eliminates manual setup
• Self-organizing network

Fault Tolerance

When a server goes offline:

graph TD
    A[Server Offline Detected] --> B[Mark Peer as Disconnected]
    B --> C[Remove from Active Sessions]
    C --> D[Continue with Remaining Peers]
    D --> E{Server Returns?}
    E -->|Yes| F[Beacon Detected]
    F --> G[Re-establish Connection]
    G --> H[Sync State Changes]
    E -->|No| I[Wait for Beacon]
    I --> E

Cross-Server Data Flow

Peers enable powerful cross-server automation:

Example: Distributed Temperature Monitoring

Server A (Living Room)
  └─> Temperature Data Point
      └─> Updates synced to peers

Server B (Bedroom)
  └─> Receives temp update from Server A
      └─> Trigger: High Threshold
          └─> Executor: Control local AC

Server C (Central)
  └─> Compute: Average all room temperatures
      └─> OutgoingWebHook: Dashboard update

Integration with TargetingNodeMetaData

Executors with TargetingNodeMetaData can target Data Points on any peer:

• Calculation: Read from Server A, write to Server B
• Compute: Aggregate data from multiple servers
• Logic Gates: Inputs from different servers

Session Management

The PeerSessionManager handles all peer connections:

• Session Tracking: Maintains list of active sessions
• TTL Cleanup: Removes expired/inactive sessions
• Connection Health: Monitors heartbeats and connection state
• Reconnection: Handles automatic reconnection logic

Best Practices

• Network Stability: Use reliable local network infrastructure
• Consistent Versions: Keep all servers on same Krill version
• PIN Security: Choose a non-obvious PIN and reset it if compromised
• Resource Planning: Consider CPU/memory across distributed load
• Monitoring: Monitor peer connection health
• Backup Strategy: Plan for server maintenance windows

Comparison: Peer vs External Server

Feature	Peer	External Server
Discovery	Automatic (beacon)	Manual configuration
Network	Local network only	Any network/internet
Latency	Very low	Variable (network dependent)
Setup	Same cluster PIN	Requires address and PIN
Trust	Beacon TOTP + shared PIN-derived token	PIN-derived Bearer token

Troubleshooting

Issue	Cause	Solution
No peers discovered	Firewall blocking UDP	Open beacon port
Intermittent connections	Network congestion	Check network health
Sync delays	High latency	Verify network performance
Authentication failures	Mismatched cluster PIN	Verify all servers use the same PIN; reset with krill-reset-pin
Beacon not validated	Wrong PIN on one server	Re-enter correct PIN on the misconfigured server

Network Requirements

• UDP Port: Open for beacon broadcasts (discovery)
• TCP Port: Open for HTTPS connections (data)
• Same Subnet: Servers should be on same network subnet for discovery
• Low Latency: Local network recommended for real-time sync

The Decentralized Mesh Network architecture is the foundation of Krill’s resilient, scalable automation platform, enabling multiple servers to work together as a unified system.

---

Last verified: 2026-04-03