Self-hosted AI: how to own your AI stack

Self-hosted AI means you run the whole thing — the models, the data they touch, and the automation built around them — on infrastructure you control, instead of renting it as a service from a vendor's cloud. A self-hosted LLM is the core of that: a language model that does its inference inside your own network, so your prompts and documents never leave the boundary you set. This is a practical guide to what a self-hosted AI stack is actually made of, the two honest ways to own one, and what "own your stack" has to include before the phrase means anything.

The demand is not abstract. If you work under a regulator, a security review, or a customer contract, "the vendor handles that for us" is a harder answer to defend every year — and self-hosting is what turns AI from something you subscribe to into something you can put in an audit.

Three words get used as if they were the same thing. They are not.

• Cloud AI runs on the vendor's servers. You send data out on every call. Fast to start; you control the least.
• Self-hosted AI runs on servers you control — your datacenter, your private cloud, or an environment fully inside your own network. The data stays on your side of the boundary.
• Local AI is the same idea shrunk to one machine: a model on a single laptop or workstation. Good for one developer; it does not give a team a shared, governed system.

Self-hosted is the version a whole organisation can run, govern, and prove. The rest of this guide is about that version.

"Self-hosted AI" is not one product you install. It is a set of parts that have to work together — and each part is a thing you either own or rent. Knowing the parts is how you avoid buying "self-hosted" that quietly phones home.

• A self-hosted LLM (inference). The model, and the server that runs it, inside your network. You choose the model, you choose when it changes, and inference happens on your hardware — so a provider deprecating a model overnight does not silently change how your system behaves.
• Orchestration. The layer that turns a model into work: chaining steps, calling tools, deciding what happens next, keeping a human in the loop where it matters. A raw model answers; orchestration is what runs the job.
• A knowledge and data layer. Where your documents, embeddings, and context live. If this sits in a vendor's cloud, "self-hosted model" buys you very little — the data path is what actually decides whether your information leaves.
• Identity and access control. Who is allowed to do what. Role-based access control (RBAC), tied to the identities your organisation already manages.
• An audit trail. A durable, exportable record of what the system did, kept where you keep your records — so "who did what, and on whose authority" is a question you can answer from your own logs.
• A sandbox and outbound control. A place for tool code and agent actions to run safely, and clear visibility of every connection that leaves your network — so nothing reaches out except the endpoints you configured.

Own all six and you own your stack. Own three and rent the other three, and you own a demo.

There are two real routes to a self-hosted AI stack. Both are legitimate; they cost you in different currencies.

Route one: assemble it yourself. Take a model server, an orchestration framework, an observability tool, a sandbox, a model proxy, an identity layer, and a trigger/scheduler — seven separate open components — and wire them together. You get maximum flexibility and you owe maximum glue. That means seven upgrade cycles, seven auth models, and an engineer whose job is keeping the seams from splitting. For some teams that trade is worth it. Go in knowing you are hiring for the glue.

Route two: run one governed runtime. Adopt a single self-hosted platform that already brings orchestration, the sandbox, identity, the audit trail, and model choice together, and point your own model and data at it. You give up some à-la-carte freedom; you get one thing to upgrade, one permission model, and one place your operations team reads. The parts still belong to you — they just arrive assembled and governed instead of as a kit.

The choice is not "build vs. buy your independence." Both routes are self-hosted; both keep the data on your side. The choice is how much integration work you want to own yourself versus how much you want to arrive already fitted together.