What actually runs on the box

---

Why this page exists

"An on-premise AI system" is a phrase doing a lot of work. It could mean almost anything, and vendors rely on that. So here is the actual contents, component by component, with the reasoning attached. If you are being quoted for something like this by anyone, this is a reasonable list to hold their proposal against.

None of it is exotic. That is rather the point: the components are open, well-understood, and replaceable, and the value is in the assembly and the ongoing care rather than in any single piece.

1. The model

One open-weights model does the general work. Today that is gpt-oss-120b, and the reason is a technical detail worth understanding, because it is the difference between a machine that works and one that does not.

It is a mixture-of-experts model: nominally 120 billion parameters, but only around five billion of them activate for any given token. That matters because a machine like this is limited by memory bandwidth, not memory capacity. A conventional dense 70-billion-parameter model will load happily onto the same hardware and then produce text more slowly than you can read it. It fits, and it is useless.

This is the single most common way to be sold the wrong machine. A spec sheet will tell you how much memory it has. It will not tell you how fast it can move that memory around, and that is the number that decides whether anyone in your office will actually use the thing. Ask for measured tokens per second on the specific model you will be given. Not benchmark scores, not parameter counts. A measurement, on that hardware, on that model.

A second, smaller model usually sits alongside it for the fast work: extraction, classification, the jobs that need to feel instant rather than clever.

How far behind these models actually are, and on which tasks, is measured in the frontier–local gap. The short version is that they reach parity on most of what an office asks and fall clearly behind on the hardest reasoning.

2. The serving layer

Something has to hold the model in memory, accept requests, and queue them when several people ask at once. In practice this presents the same interface that OpenAI's API uses, which sounds like a footnote and is in fact the most important architectural decision in the whole system.

It means every tool built on top is pointed at an address, and that address can be a machine in your building or a hosted endpoint somewhere else, with nothing else changing. You are not locked into the box by having bought the box. If a hosted model turns out to be better for one particular job, that job can use it, and the decision is a line of configuration rather than a rebuild.

3. Retrieval over your documents

Your files are broken into passages, indexed, and searched when a question arrives. The matching passages go to the model, which answers from them and cites which document it used. This is retrieval, and it is not the same thing as training a model on your documents, though the two get asked for interchangeably. Retrieval or training is the distinction, and it decides most of the budget.

Two things about this are harder than they look, and they are where the effort actually goes.

The first is permissions. A retrieval system that ignores who is asking will eventually quote a confidential file to someone who should not see it, and it will do so helpfully and in good faith. Access control has to live in the retrieval layer, not be bolted on afterwards.

The second is staying current. An index built once is wrong within a month. Documents change, get superseded, get withdrawn. A system confidently citing a version of a policy you replaced in March is worse than no system, because it is trusted.

4. The interface

A chat window, reachable from a browser on the office network. It looks like the tools your staff already use, because there is nothing to gain from novelty here. An interface people have to be trained on is an interface people stop opening.

Behind it: separate conversations per person, a shared document library, and a record of who asked what. That last one is not surveillance, it is how you answer the question "has anyone put a client's file into this" six months from now, at a point when you need a real answer rather than a reassurance.

5. The tools that do the actual work

This is where the value is, and it is the least generic part of the system. A chat window is a nice thing to have and does not change how a firm operates. The tools are what does: the one that turns field notes into a formatted document, the one that reads the job inbox, the one that produces the monthly report that somebody currently assembles by hand.

Each is specific to one workflow, and each is built after watching the workflow rather than before. That specificity is the whole reason it gets used.

6. Your adapter, if you have one

A few megabytes of trained weights that clip onto the base model and make it write in your house style. It is loaded on demand and can be switched off. Whether you need one at all is a separate argument. The short version is that it changes how the system writes, never what it knows.

What is deliberately not on the list

No outbound path to any AI provider. For a locally-installed system this is a firewall rule you can inspect rather than a promise in a contract, which is the entire argument in why your AI should live in your office. It is also the thing worth verifying yourself, or having your IT people verify, rather than accepting on trust, including from me.

No telemetry, and no usage data leaving the building. This includes mine. I cannot see what your staff ask it.

No autonomous agents running unsupervised. Long chains of automated reasoning are the least reliable thing these systems do, and errors compound at every step. Three or four dependent steps is comfortable. Thirty is a demo. Anything that touches a document your name goes on has a person in the loop, deliberately.

What it looks like when it breaks

Worth saying, because every honest inventory should include this.

The most common failure is dull: an update to something in the chain changes behaviour slightly and the output quality drifts. Nobody notices for a fortnight. This is most of what the monthly fee is actually for: not fixing outages, but noticing drift.

The second most common is a document format nobody anticipated. A scanned PDF with no text layer, a spreadsheet used as a form, a file produced by software from 2009. These fail visibly, which is the good kind of failure.

The rarest and worst is a confident wrong answer that passes review because it looks exactly like a right one. That risk never goes to zero. It is why the review step exists on anything consequential, and why I would not sell you a system that removes it.