Thirty Models, One Box, Zero Cloud

I run 30 models on one Mac. No cloud. Not a single API call leaves the box.

That number sounds like a flex, and the first instinct is to read it as a capacity story — how many models can you cram onto consumer silicon before it falls over. That's the boring version. The interesting version is the opposite: 30 models exist not because I wanted a zoo, but because no single model is the right tool for every job, and I stopped pretending one could be.

The whole fleet runs on Ollama 0.30.8 on an Apple M4. Embeddings, code review, overnight reasoning, fast structured output, voice — all of it served locally, all of it free of a metered token bill. That constraint isn't a hair shirt. It's the thing that forced the discipline I actually want to talk about.

The Lazy Version of "Use AI"

The default way people wire up AI in a system is to pick one strong model and route everything through it. It's understandable — one endpoint, one prompt style, one mental model. If it's good at code, it's probably good enough at JSON, and good enough at summarizing, and good enough at embeddings. Right?

That's how you end up paying a reasoning model's latency to extract three fields from a string. It's how you hand a deliberation task to a model tuned for throughput and watch the quality fall off. "Good enough at everything" is a polite way of saying "wrong for most things." A generalist routed everywhere is the path of least resistance, and least resistance is rarely where the good engineering lives.

I learned this the hard way, on a worker that failed green — a job scorer built on a reasoning model in thinking mode that spent its entire budget reasoning and never emitted the JSON I asked for. The fix wasn't a better prompt. It was a different model class. That bug taught me the rule the whole fleet now runs on: match the model to the task.

The Actual Strategy

So here's how the work is actually split. Four models do the heavy lifting, each pointed at the job it's genuinely best at:

• forge-gemma4-12b handles code review and local vision. It's the biggest hammer in the box, and I reserve it for the jobs that need the weight — reading a diff and reasoning about it, or looking at an image and describing what's there. This is the one place I want a model that thinks hard.
• qwen3.5:9b does overnight reasoning. When the chain runs unsupervised at 5 AM and needs to actually deliberate — synthesize research, reason across a multi-step problem — this is the model I trust to take its time and get somewhere.
• gemma4:e4b is the fast path: structured JSON and voice. This is the workhorse for scoring, classification, extraction — anything where I want a shaped answer back now, not a reasoning trace. It's the model that replaced the one that was thinking too much.
• nomic-embed-text does embeddings, and nothing else. It's frozen on purpose. Embeddings are the foundation of every vector search in my memory layer, and the one thing you do not do is quietly swap the model that generated your existing vectors. Freeze it, and everything downstream stays comparable.

On top of those, I run 5 custom LoRA adapters — argus, chisel, conductor, ember, herald — each a small, task-shaped specialization layered onto a base model instead of a whole new set of weights. That's the cheap, sharp end of the same idea: when a task is narrow enough, you don't need a new model, you need a small adapter that bends an existing one toward the job.

Code review wants weight. Scoring wants speed. Reasoning wants patience. Embeddings want stability. Those are four different demands, and pretending one model satisfies all four is how you get a system that's mediocre at all of them.

Benchmark, Don't Guess

Here's the part that matters more than the lineup, because the lineup will change and this won't.

I didn't arrive at this split by reading model cards. Model cards tell you what a model is good at in general, on someone else's benchmark, against someone else's task. None of that is your task. The only thing that tells you which model wins on your job is running your job through every candidate and looking at the output.

Picking a model for its reputation is the single most common way I see AI systems go wrong. A model has a great launch, the benchmarks look incredible, and it gets slotted in as the answer to everything — including the things it's quietly bad at in your specific pipeline. The reputation is real. The fit is assumed. And assumed fit is where systems rot.

So I benchmark. New model drops, I don't swap it in because the leaderboard says so — I run it against the actual task it's a candidate for, with real inputs, and compare the output to what I'm already running. Sometimes the shiny new model wins. Sometimes the older, smaller, faster one I already have is better for this job, and the new model is a great general model that happens to be wrong here. You only know which by measuring. There's a worker in my overnight chain whose entire job is scouting new models — and even it doesn't get to promote a model on hype. The bar is: prove it on the task, or stay on the bench.

Guessing feels faster. It costs you later, in a worker that fails quietly or a pipeline that's subtly worse than it could be, and you won't know why because you never measured the alternative.

Why Zero Cloud Sharpens This

The no-cloud constraint isn't just a privacy or cost decision, though it's both. It's a forcing function for the discipline above.

When every model lives on your own hardware, "just use the big API model for everything" stops being an option. You can't paper over a bad model-task fit with someone else's frontier model and a credit card. You have a finite box, a fleet of specialists, and the requirement to point each task at the right one. The constraint makes the lazy path unavailable, and removing the lazy path is how you end up doing the work properly.

It also means every one of those 30 models is mine — I can fine-tune it, freeze it, layer an adapter onto it, swap it, benchmark it, without asking anyone's permission or watching a meter spin. The fleet is a deliberate, owned, measured set of tools, not a single rented hammer I treat every problem as a nail for.

The Real Lesson

Thirty models on one box is a number that grabs attention, but it's not the point. The point is what the number is evidence of: a system where the question is never "which model is best," it's "which model is best for this" — and where that question gets answered by running the task, not by reading the hype.

Match the model to the task. Benchmark, don't guess. Freeze the things that need to stay stable, specialize the things that need an edge, and never let a model's reputation do your measuring for you. That's the whole strategy, and it scales from one model to thirty without changing shape.

If you're building local AI systems and want to talk model strategy, I'm at reed@grainlabs.io.