What happened

An LLM node pointed at qwen2.5vl:32b (a vision model) on a 2×RTX 5090 box wedged the GPU at 100% for minutes with no error surfaced anywhere. ollama.service logs showed the load request retrying forever, shedding one GPU layer each attempt:

1
2
3
4
load request="{... KvSize:128000 ... GPULayers:49 ...}"
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 53094.90 MiB on device 0: cudaMalloc failed: out of memory
load request="{... KvSize:128000 ... GPULayers:48 ...}"
...

The node itself sat in USER_EDIT/NONE with an empty snapshot — indistinguishable from a hang to an operator watching the UI.

Root cause

ServerLLMProcessor.buildRequestBody() sent Ollama a request with no options block at all. Without an explicit num_ctx, Ollama falls back to the model’s Modelfile default — 128K for the Qwen-VL family — and tries to allocate a KV cache sized for that context on every load. At 128K context the cache alone needs ~50 GB, more than the box’s usable VRAM, so the load OOMs and Ollama’s degrade-and-retry loop runs indefinitely. testConnection() (the “test connection” ping used from the node’s UI) had the same gap, so even a connectivity check could trigger the same OOM on a cold model load.

The demo only ever worked because it happened to point at a hand-baked ollama create ... PARAMETER num_ctx 8192 variant of the model — a correctness requirement pushed onto the operator, entirely out-of-band from krill, with no error if they forgot it.

Fix

ServerLLMProcessor now sends options.num_ctx: 8192 on every Ollama request (both the real inference path and testConnection) via a shared ollamaOptions() helper. 8192 is a size every krill workload today comfortably fits in, and it is far below any GPU’s VRAM budget on the hardware this runs on — no more silent 128K default.

This is a hardcoded default, not a per-node setting, because the field to make it per-node-configurable (numCtx on LLMMetaData) lives in krill-sdk, which is owned by krill-oss, not this repo. Making it configurable is tracked upstream; this fix stops the OOM unconditionally in the meantime, with no SDK release in the path.

The second half of the original report — put a timeout on the Ollama call and route failures to NodeState.ERROR instead of leaving an empty snapshot — was already true of the code: callBackend() already carries a 5-minute request/socket timeout and calls nodeManager.failed() (which sets NodeState.ERROR) on any exception or non-2xx response. No change was needed there.

Prevention

• A backend request with no explicit resource limits inherits the backend’s default, not krill’s. Anywhere krill posts to an external inference/compute backend, treat every tunable the backend exposes as “krill’s responsibility to set,” not “safe to omit.” An omitted field is a silent inheritance of someone else’s default, and that default can be sized for a use case krill never intended (e.g. a Modelfile tuned for a 128K-context research workload, not a home-automation prompt).
• A hang and a crash can look identical to an operator. The GPU thrashing at 100% for four minutes was indistinguishable from a normal slow inference in the node’s state. Any external call that can degrade instead of failing cleanly (retry loops, backoff, partial loads) needs either a hard timeout or a way to observe “this is unusually slow” from the node’s error surface — a bound at the edge is what turns an opaque hang into a diagnosable error.