Local AI infrastructure with a control plane.

What it does

• Vision — every image-bearing request lands here first
• Ring camera snapshots
• Frigate events
• Screenshot understanding
• Fast classifier: the “is this even worth escalating?” pre-filter
• JSON utility: structured outputs for the camera pipeline, notifications, and tagging

Latency

p50 target

≤ 1 s

p95 target

≤ 2 s

p50 measured

0.61 s

p95 measured

0.81 s

Check

4 / 4 routing safety cases pass

Check

Zero false suppressions

Check

Zero invalid JSON

Optimizations

• AWQ INT4 quantization — one-fifth the weights of FP16
• vLLM with continuous batching and prefix caching
• Production context locked to 32K tokens after a latency / KV / VRAM sweep
• Auto-start as a user-level systemd unit — no manual login after reboot
• Approximately 9.5 GB of 16 GB used, leaving headroom for CUDA-graph capture and prefix-cache growth

Hard rules

• Nothing else co-hosts on the 4090 — no coding model, no large-context model
• The slot is single-purpose on purpose

What it does

• Code generation
• Refactoring
• Multi-file edits
• Agentic tool-use work
• Cursor and OpenCode are routed here by default
• Longer-context utility work

Optimizations

• AWQ W4A16 quantization using compressed tensors
• TurboQuant KV cache in k8v4 mode — 8-bit keys, 4-bit values
• ≈ 2.6× KV compression vs FP8 with negligible quality cost
• ≈ 14% more KV cache slots than the FP8 baseline at the same context budget
• Per-layer aware: 42 quantized layers use TurboQuant; 4 boundary layers auto-skip to FLASH_ATTN for quality safety
• 128K-token context window
• Auto-start as a user-level systemd unit with --profile long

Hard rules

• No cross-GPU tensor parallelism with the 4090 — Thunderbolt latency would kill it
• No model swapping — treat it as a single resident endpoint
• No CPU offload — 96 GB is plenty of headroom and any swapping would gut throughput

What it does

• Deep reasoning
• Architecture review
• Planning
• Hard debugging
• Final review and risk analysis
• Safety fallback for every other route

What it does not do

• Spark never falls back — it is the bottom of the chain
• If Spark is down, the router returns a 503, period
• No silent downgrade to a less-capable model on a safety-critical decision

Optimizations

• FP8 weights — throughput and KV-cache efficiency on GB10’s unified memory
• vLLM with prefix caching
• Chunked prefill
• YaRN for long-context overrides
• Open WebUI front-end pinned to its own data directory so other UIs do not collide
• Treated by the router as a network-attached upstream over the private overlay

Hard rules

• Spark never falls back to another model
• If Spark is down, the answer is 503 — never a quiet downgrade
• The router does not try to start, stop, or recover Spark — the operator gets paged

What it does

• General chat
• Light Q&A
• Rewrites, format cleanup, paraphrasing, translation
• Medium-context summarization (≈ 1K–8K tokens)
• Text-only spillover when ubuntu-4090 is saturated
• Code spillover when ubuntu-6000 is busy

What it does not do

• No image work — Gemma cannot see, so the router never sends image-bearing requests here
• Images always go to ubuntu-4090 or, during an outage, to Spark
• No final architecture review or deep reasoning — that stays on Spark
• No camera safety decisions

Optimizations

• vLLM-style serving via LM Studio with three concurrent slots
• Router concurrency cap matched exactly to the upstream slot count
• Third caller gets a `busy` response
• Fourth caller is sent to the fallback instead of queueing forever
• gpu-memory-utilization set to 0.86
• 32K context
• Batched-token budget tuned to the 32 GB card
• Inbound traffic pinned to the private overlay network at the Windows Firewall layer

Hard rules

• SSH, the LLM endpoint, and the Netdata UI are unreachable from public Wi-Fi by design
• Never the image path — the router carries a has_image flag through preflight so this rule wins per request