What happened

The SDK has modelled Ollama’s multimodal field since the LLM node existed:

1
2
/** Base64-encoded image payloads for multimodal models. */
val images: List<String> = emptyList(),

Nothing ever set it. buildUserMessage() only ever constructed Message(role, content), so a vision model configured on an LLM node could receive text and nothing else. There was no way for any node in the swarm to hand it a picture.

This blocked the “Local LLM Nodes” demo. The Coop Camera DataPoint held a written description of the frame — what a vision model would have said — and the real photo was composited into the video as a corner overlay. Both LLM calls in that demo are genuine inference; only the eye was fake.

Root cause

Two independent gaps, and neither alone was sufficient:

1. Nothing populated Message.images (this issue).
2. Nothing could reach an image to populate it with. ServerCameraProcessor captured a JPEG and never published its path, so an LLM wired to a Camera received CameraMetaData carrying its config — resolution, enabled — and an empty snapshot (krill#842, fixed first).

A field existing in the data model reads as a supported feature. images was defined, documented, serialized under the right @SerialName, and dead. Nothing failed; the model simply never saw anything.

Fix

ServerLLMProcessor.imagesFrom() base64-encodes the current frame of every Camera node wired into the LLM’s inputs, and buildUserMessage() attaches them to the user Message plus a line telling the model to describe what it actually sees rather than infer the scene from the node JSON.

Wiring a Camera into an LLM’s inputs is the opt-in — deliberately no separate meta flag, which the issue had proposed. A user who connects a camera to a model is asking the model to look at it. A flag would have to default to off (or it changes existing behaviour), so the feature would ship invisible; and if the configured model has no vision capability, the backend says so and the error lands on the node — a clearer signal than a silently-off flag. It also keeps the change entirely inside krill, with no SDK release in the path.

Frames not on local disk are skipped with a warning, not sent. A camera on a peer publishes a path into that host’s filesystem, which means nothing here.

Prevention

• A field in the data model is a promise. An unpopulated one is a lie. Message.images looked like a supported feature from every angle — typed, documented, correctly serialized — and was never written to. When adding a field to a wire model, add the code that fills it in the same change, or the field is worse than absent: absent fails loudly at the call site, dead reads as done.
• Grep for writes, not just reads, when auditing a capability. images had exactly one reference in the whole repo: its own declaration.
• Two gaps can hide each other. Populating images was pointless while no node could name an image file, and publishing the camera’s path was pointless while nothing read it. Either one alone looks like it “doesn’t work”, and the demo’s workaround (fake the camera with a text description) made both survive. When a feature is “not possible”, check whether it is one missing piece or a chain of them.
• A demo workaround is a bug report. The team wrote a textual description of what a vision model would have returned and composited the real photo in during editing. That is a precise, unambiguous statement of a missing capability — it just wasn’t filed as one until later.