What Still Rents: The Portfolio Case for Local AI

New to the series? Start with Part 0: Same Weights, Different Product.

The question a leader actually asks

Strip away the benchmark rankings, and the decision on the table is simple:

Do we buy the hardware, or keep paying per token forever?

That is the question I get from peers running AI inside real organizations. It is not "is local better than cloud." It is a capital-versus-operating-expense question with privacy, latency, and control riding along on top. This post answers it with the evidence from the previous two parts, and with a rule you can defend.

The rule: own the substrate, rent the frontier. Own the capability you use constantly, that touches sensitive data, and that you want to iterate on without watching a meter. Rent the capability you need rarely, that has to be best-in-class, or that you simply do not want to operate.

What open weights threaten, and what they don't

Harnessed local models do not make the cloud obsolete. They challenge default rent for a surprisingly large slice of real work.

What is threatened

• Default token rent for internal knowledge work. Drafting, summarizing, extraction, reasoning over internal documents, offline copilots, privacy-first workflows. This is the bulk of enterprise AI usage, and it is exactly where a harnessed local model is already good enough.
• Privacy as a premium feature. Local inference makes "no vendor transcript" the default for a class of work, not an upsell.
• Metered iteration. When the GPU is on your desk, you can afford more "try, revise, tighten" loops without pricing every one.

What is not threatened

• Polish as the job. SVG beauty contests, brand tone, UI finesse, "make it delightful." Frontier still wins here.
• Hard SWE depth. Deep refactors, tricky concurrency, novel debugging across unfamiliar codebases, and anything where tool integration and verification are the actual work.
• Newest capability first. When the only thing that matters is what shipped in the last thirty days, or what only a frontier lab has, you rent it.
• Operations you do not want to own. Uptime, scaling, observability, incident response, access control, and vendor legal posture can all be worth paying for.

This is why the right posture is not tribal. It is a portfolio.

Ownership economics (evidence-backed, and sensitive)

The point of this section is not to win an argument. It is to make a clean decision when someone asks whether to buy hardware or keep renting.

The communicated "~$6K desk"

The honest version, not MSRP fantasy but the disciplined prebuilt or allocation path:

The line I would actually say out loud: a complete 5090 desk lands around six grand if you buy a prebuilt or already own the chassis. Buying the GPU alone at street can cost almost that.

Throughput basis: baseline and optimized

The original economics used a deliberately conservative planning rate:

• Phase 1 measured decode: 53.9 tok/s on Q4_K_S
• Original planning case: 60 tok/s
• Optimized real-task planning case: 140 tok/s, the midpoint of the later 138–145 wall tok/s HTML/SVG range
• Optimized count probe: 168 tok/s, excluded from TCO because it is unusually easy for MTP

Part 4 documents the later llama.cpp, NVFP4, q4-KV, and MTP stack. The important distinction is capacity versus consumption. A 140 tok/s system can process more tokens; it only saves more money when people or agents use them.

Light, power, heavy (capacity ceiling)

Light means occasional chat. Power means daily agent and rewrite work. Heavy means automated loops that plan, call tools, retry, and rewrite for hours.

The API shadow bill (illustrative)

To keep this honest, the API math remains deliberately simple and list-like:

• Blended proxy: ~$12 / 1M tokens (example mix: 30% input at $5 plus 70% output at $15)

The original power-user case implied roughly 1.5–2 years of equivalent consumed volume before privacy and latency. If a sustained agent workload actually consumes the optimized power-user capacity, the same arithmetic can move below one year. That is a scenario, not a forecast. Idle throughput has no economic value.

If your token mix is more input-heavy, your negotiated prices are lower, or your usage is genuinely light, cloud wins for longer. That is fine. The decision is not moral. It is arithmetic.

The Heavy row is not “someone chatting a lot.” It is automation. Point an agent runner at the desk, the kind of loop that plans a task, calls tools, checks its own output, and rewrites until it passes, and it will generate and consume tokens on both sides of the meter for as long as you let it run. Light interactive chat rarely justifies the hardware on pure dollars. Sustained agent automation can, which is why the ownership question is really a question about how much of your work is about to become unattended.

When cloud still wins (and why that is fine)

Cloud wins when you are paying for more than tokens.

• Polish and aesthetics dominate. SVG, design-heavy copy, brand voice, UX detail. Frontier wins most of these beauty contests.
• Hard SWE is the actual task. Deep debugging, complex architecture, multi-repo changes, correctness work where "seems right" is a failure mode.
• Reliability is the product. You want SLAs, autoscaling, rate limiting, audit trails, and incident response your team did not have to build.
• You need the newest capability. Novel reasoning behavior, best-in-class tool use, or frontier features open weights have not caught up to yet.
• You cannot justify owning the ops. Driver patches, serving stacks, access control, internal distribution, and "who owns the pager" questions.

One scope note: the cloud comparison across this series is OpenAI only, by design. GPT-5.2 was the primary control, with GPT-4.1, GPT-5.5, a cheap-tier Luna, and a two-year-older GPT-4o run for contrast on the same prompts. That is a deliberate local-versus-OpenAI read, not a claim about every frontier vendor.

A routing table you can defend

If you want a default posture that will not embarrass you in six months, it looks like this:

Two meta-rules keep this sane:

1. Do not route by ideology. Route by risk and unit economics.
2. Owning a GPU is not owning an AI system. The harness, posture, schemas, evals, access control, and operational care, is what turns hardware into a durable asset. That is the Harness Principle from Part 1, pointed at strategy instead of a single prompt.

Limitations

The tables above are a routing aid, not a universal constant. Payback is sensitive to actual utilization, token mix, and the rates you pay. The 60 tok/s column preserves the original conservative model; the 140 tok/s column uses the midpoint of the optimized HTML/SVG wall range, not the 168 tok/s count probe. A native 262K allocation fit, but the longest prompt tested was approximately 48K. And this remains Qwen3.8-27B, a mid-size open model, not Qwen3.8-Max. Read the arithmetic as a method, then run it on your own inputs.

Closing

The conclusion I am comfortable with is narrow.

The assumption worth rejecting is that rented frontier chat is the only place intelligence lives for everyday knowledge work.

Open weights plus commodity GPUs do not end the cloud. They end the assumption that renting is always the adult choice. Frontier labs still matter, they are where genuinely new capability appears first and where the hardest research concentrates, and the move is not to pretend a box under a desk replaces them.

The move is to stop renting by default when ownership is already good enough, and materially cheaper, for a large class of real work. Own the substrate. Rent the frontier. And remember that on either one, the harness is the product.

Method pointers

• Series hub: Qwen3.8: Same Weights, Different Product
• Serving-stack capstone: 262K on 32 GB
• Demos: play the one-shot apps
• Follow: linkedin.com/in/brianletort