Symptom

Ghost’s list_nodes on first contact after a “fresh” restart returned pre-existing nodes, including some whose IDs matched fixtures created during a prior verify run weeks earlier. krill.service’s ActiveEnterTimestamp confirmed a genuine restart running the new PR’s build, so the binary/restart machinery worked — only the database state failed to reset. First observed and root-caused on the krill-oss copy of the same workflow (Sautner-Studio-LLC/krill-oss#262 / #263); this repo’s .github/workflows/Verify Agent Ghost.yml had byte-identical content and the same bug.

Root cause

/srv/krill/data is created 0750 krill:krill by server/package/DEBIAN/postinst (install -d -o krill -g krill -m 0750 /srv/krill/data), and postinst’s adduser calls only ever add the krill system user itself to supplementary groups — the ghost runner’s own user is never added to the krill group. The wipe step’s sudo -n rm -f /srv/krill/data/*.db expands that glob in the calling, unprivileged shell before sudo ever runs (sudo only elevates the exec’d command, not the shell that built its argv). Against a directory the caller can’t read, the glob silently matches nothing, rm -f receives the literal nonexistent filename *.db, and -f suppresses the resulting error. The step logged success and deleted nothing — old DB state survived an ostensibly “fresh” restart.

Fix

• .github/workflows/Verify Agent Ghost.yml: wrap the delete in sudo -n bash -c 'rm -f /srv/krill/data/*.db' so the glob expands inside a root-invoked shell that can actually read the directory, then assert (via a nullglob count) that nothing matching survived, hard-failing the step with an ::error:: annotation if it didn’t.

Prevention

• A sudo rm -f <glob> over a directory the calling shell can’t read is not evidence the files were removed. The glob expansion happens in the caller’s shell, not under sudo; -f exists specifically to suppress “no such file” errors, which means it also suppresses the signal that the glob never matched anything. Any privileged deletion of files the unprivileged caller can’t list should expand the glob (or otherwise enumerate the target) inside the sudo‘d shell — sudo -n bash -c 'rm -f /path/*.ext', not sudo -n rm -f /path/*.ext.
• A wipe/reset step meant to establish a known-empty precondition should assert the postcondition, not just run the delete and move on. Counting remaining matches after the delete (as this fix does) converts “we ran a command that we believe deletes things” into “we verified nothing matching survived.”
• Identical workflow files copied across sibling repos carry identical bugs. This repo’s Verify Agent Ghost.yml and krill-oss’s derive from the same original commit; when a bug is found and fixed in one, check the other copy before assuming it’s a one-repo issue.