Post

Icon TLS Certificate Management

Understanding how Krill servers and clients manage TLS certificates for secure peer-to-peer communication

Posted
By Benjamin Sautner
7 min read
TLS Certificate Management

Overview

Krill uses a trust-on-first-use (TOFU) model with self-signed TLS certificates to secure peer-to-peer communication between servers and clients on your local network. This document explains how the certificate system works, how clients establish trust, and how you can customize or replace certificates.

Architecture

Certificate Flow Diagram

sequenceDiagram
    participant Client as Krill Client
    participant Server as Krill Server
    participant TrustStore as Trust Store

    Note over Server: Server starts with<br/>self-signed certificate

    Client->>Server: Discovery (mDNS/Beacon)
    Server-->>Client: Server URL announced

    Client->>Server: GET /trust (insecure)
    Note right of Client: Uses trust-all TLS client<br/>to fetch certificate

    Server-->>Client: Returns krill.crt

    Client->>TrustStore: Store certificate<br/>as {hostname}.crt

    Client->>Server: HTTPS requests (secure)
    Note right of Client: Uses stored certificate<br/>for TLS verification

Trust Establishment Process

flowchart TD
    A[Client Discovers Server] --> B{Certificate in Trust Store?}
    B -->|Yes| C[Use Stored Certificate]
    B -->|No| D[Fetch Certificate via /trust]
    D --> E{Certificate Valid?}
    E -->|Yes| F[Store in Trust Store]
    E -->|No| G[Log Error, Skip Server]
    F --> C
    C --> H[Establish Secure Connection]
    H --> I[Download Host Node Data]
    I --> J[Connect via MQTT/WebSocket]

Server-Side Implementation

Certificate Generation

When you install Krill Server via the Debian package, the postinst script automatically generates a self-signed TLS certificate during first installation:

Certificate Files Location: /etc/krill/certs/

FileDescriptionPermissions
krill.crtX.509 certificate (public)644
krill.keyRSA private key600
krill.pfxPKCS12 keystore for JVM644
.pfx_passwordKeystore password600

The certificate is generated with:

  • Algorithm: RSA 2048-bit with SHA-256 signature
  • Validity: 10 years (3650 days)
  • Subject Alternative Names (SANs):
    • localhost
    • {hostname}
    • {hostname}.local
    • {host-ip}

Password Security

Each installation generates a unique, cryptographically strong password for the PKCS12 keystore:

1
2

# Password is generated using OpenSSL's cryptographic random generator
PFX_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32)

The password is stored securely in /etc/krill/certs/.pfx_password with restricted permissions (mode 600), readable only by the krill service user.

Server TLS Configuration

The Krill server loads certificates at startup from KtorConfig.kt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

// Server reads password from secure file
val password = readPfxPassword()

// Loads certificate and key from PEM files
val keyStore = loadKeyStoreFromPem(
    certFile = File("/etc/krill/certs/krill.crt"),
    keyFile = File("/etc/krill/certs/krill.key"),
    password = password
)

// Configures SSL on port 8442
sslConnector(keyStore, "krill", { password }, { password }) {
    port = 8442
}

Trust Endpoint

The server exposes its certificate via the /trust endpoint for clients to download:

1
2
3
4
5
6
7
8

get("/trust") {
    val file = File("/etc/krill/certs/krill.crt")
    if (file.exists()) {
        call.respondBytes(file.readBytes(), ContentType.Application.OctetStream)
    } else {
        call.respond(HttpStatusCode.NotFound, "Certificate not found")
    }
}

Client-Side Implementation

Trust Store Locations

Clients store downloaded certificates in platform-specific locations:

PlatformTrust Store Location
JVM/Desktop/var/lib/krill/trusted/
Android{app_files}/trusted/
iOSNot yet implemented
Web/WASMBrowser handles TLS

Certificate Download Process

When a client discovers a new server, it uses a temporary “insecure” HTTP client to download the certificate:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// Insecure client to fetch certificate
val insecureClient = HttpClient(CIO) {
    engine {
        https {
            trustManager = object : X509TrustManager {
                override fun getAcceptedIssuers() = arrayOf<X509Certificate>()
                override fun checkClientTrusted(certs: Array<out X509Certificate>?, authType: String) {}
                override fun checkServerTrusted(certs: Array<out X509Certificate>?, authType: String) {}
            }
        }
    }
}

// Fetch certificate from /trust endpoint
val certBytes = insecureClient.get("${serverUrl}/trust").body<ByteArray>()

// Store certificate locally
File("$trustStore/${hostname}.crt").writeBytes(certBytes)

Building the Trust Manager

When making HTTPS requests, clients build a custom trust manager from all stored certificates:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

fun buildTrustManagerFromTrustedCerts(): X509TrustManager {
    val keyStore = KeyStore.getInstance(KeyStore.getDefaultType())
    keyStore.load(null)

    // Load all certificates from trust store
    File(trustStore).listFiles()?.forEachIndexed { i, file ->
        val cert = CertificateFactory.getInstance("X.509")
            .generateCertificate(file.inputStream())
        keyStore.setCertificateEntry("krill-peer-$i", cert)
    }

    val tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
    tmf.init(keyStore)
    return tmf.trustManagers.first { it is X509TrustManager } as X509TrustManager
}

Generating Custom Certificates

Replace with Your Own Self-Signed Certificate

You can regenerate certificates at any time using the included script:

1

sudo /path/to/scripts/cert

Or manually:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

# Set variables
CERT_DIR="/etc/krill/certs"
HOSTNAME=$(hostname)
HOST_IP=$(hostname -I | awk '{print $1}')

# Generate password
PFX_PASSWORD=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32)
echo "$PFX_PASSWORD" | sudo tee "$CERT_DIR/.pfx_password" > /dev/null
sudo chmod 600 "$CERT_DIR/.pfx_password"
sudo chown krill:krill "$CERT_DIR/.pfx_password"

# Create OpenSSL config with SANs
cat > /tmp/san.cnf << EOF
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = Local
L = Local
O = MyOrganization
CN = $HOSTNAME.local

[v3_req]
subjectAltName = DNS:localhost,DNS:$HOSTNAME,DNS:$HOSTNAME.local,IP:$HOST_IP
EOF

# Generate certificate and key
sudo openssl req -x509 -nodes -days 3650 \
  -newkey rsa:2048 -sha256 \
  -keyout "$CERT_DIR/krill.key" \
  -out "$CERT_DIR/krill.crt" \
  -config /tmp/san.cnf

# Create PKCS12 keystore
sudo openssl pkcs12 -export \
  -in "$CERT_DIR/krill.crt" \
  -inkey "$CERT_DIR/krill.key" \
  -out "$CERT_DIR/krill.pfx" \
  -name "krill" \
  -passout pass:"$PFX_PASSWORD"

# Set permissions
sudo chmod 600 "$CERT_DIR/krill.key"
sudo chmod 644 "$CERT_DIR/krill.crt" "$CERT_DIR/krill.pfx"
sudo chown -R krill:krill "$CERT_DIR"

# Restart Krill server
sudo systemctl restart krill

# Cleanup
rm /tmp/san.cnf

Using a CA-Signed Certificate

To use a certificate signed by a Certificate Authority (CA):

  1. Generate a Certificate Signing Request (CSR):
    1
2
3
4

    openssl req -new -nodes \
  -keyout /etc/krill/certs/krill.key \
  -out /tmp/krill.csr \
  -subj "/C=US/ST=YourState/L=YourCity/O=YourOrg/CN=krill.example.com"

  2. Submit CSR to your CA and receive the signed certificate.

  3. Install the signed certificate:
    1

    sudo cp signed-certificate.crt /etc/krill/certs/krill.crt
  4. Create the PKCS12 keystore:
    1
2
3
4
5
6
7

    PFX_PASSWORD=$(cat /etc/krill/certs/.pfx_password)
sudo openssl pkcs12 -export \
  -in /etc/krill/certs/krill.crt \
  -inkey /etc/krill/certs/krill.key \
  -out /etc/krill/certs/krill.pfx \
  -name "krill" \
  -passout pass:"$PFX_PASSWORD"
  5. Restart the Krill server:
    1

    sudo systemctl restart krill

Using Let’s Encrypt Certificates

If your Krill server is accessible from the internet, you can use Let’s Encrypt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# Install certbot
sudo apt install certbot

# Obtain certificate (standalone mode - stop Krill first)
sudo systemctl stop krill
sudo certbot certonly --standalone -d your-domain.com

# Copy certificates
sudo cp /etc/letsencrypt/live/your-domain.com/fullchain.pem /etc/krill/certs/krill.crt
sudo cp /etc/letsencrypt/live/your-domain.com/privkey.pem /etc/krill/certs/krill.key

# Create PKCS12 keystore
PFX_PASSWORD=$(cat /etc/krill/certs/.pfx_password)
sudo openssl pkcs12 -export \
  -in /etc/krill/certs/krill.crt \
  -inkey /etc/krill/certs/krill.key \
  -out /etc/krill/certs/krill.pfx \
  -name "krill" \
  -passout pass:"$PFX_PASSWORD"

# Set permissions
sudo chown -R krill:krill /etc/krill/certs
sudo chmod 600 /etc/krill/certs/krill.key

# Start Krill
sudo systemctl start krill

Upgrading from Older Versions

When upgrading from an installation that used the hardcoded password, the system automatically migrates:

  1. If certificates exist but no password file is found, the legacy password is written to the password file
  2. The server continues to work with existing certificates
  3. You can optionally regenerate certificates for improved security

To upgrade to a new certificate with a unique password:

1
2
3
4
5
6
7
8
9

# Remove existing certificates (backup first if needed)
sudo rm /etc/krill/certs/krill.*
sudo rm /etc/krill/certs/.pfx_password

# Reinstall or run the cert script
sudo /path/to/scripts/cert

# Restart server
sudo systemctl restart krill

Security Considerations

Trust Model

Krill uses Trust-On-First-Use (TOFU), similar to SSH. This means:

  • First connection: The certificate is trusted and stored
  • Subsequent connections: The stored certificate is verified
  • Certificate changes: If a server’s certificate changes, clients must re-establish trust

Network Security

  • Certificates are only valid for the local network
  • The /trust endpoint should not be exposed to the public internet
  • Consider using a VPN or firewall rules for additional security

Password File Security

  • The .pfx_password file is readable only by the krill user (mode 600)
  • Never share or commit the password file to version control
  • The password is unique per installation

Troubleshooting

Certificate Not Found

If the server fails to start with certificate errors:

1
2
3
4
5
6
7
8
9

# Check certificate files exist
ls -la /etc/krill/certs/

# Check permissions
stat /etc/krill/certs/.pfx_password

# Regenerate if needed
sudo rm /etc/krill/certs/krill.* /etc/krill/certs/.pfx_password
sudo dpkg-reconfigure krill

Client Cannot Connect

If clients cannot establish trust:

  1. Verify the server is running: sudo systemctl status krill
  2. Check the /trust endpoint: curl -k https://server-ip:8442/trust
  3. Check client trust store for stored certificates
  4. Verify the client can reach port 8442

Certificate Fingerprint Verification

To manually verify a certificate:

1
2
3
4
5

# On the server
openssl x509 -noout -fingerprint -sha256 -in /etc/krill/certs/krill.crt

# Compare with what clients receive
curl -k https://localhost:8442/trust | openssl x509 -noout -fingerprint -sha256

See Also

Documentation, Security, Krill Server
security tls certificates https encryption p2p
This post is licensed under CC BY 4.0 by the author.