At some point in most AI projects, someone asks: “Are we even allowed to send this data to an AI company?” Then someone else asks whether you could just run your own model instead. Both are fair questions. The answer depends on your data, your volume and who will keep the thing running.
This post compares three ways to use a large language model (LLM), the kind of model behind ChatGPT: a public API, an enterprise API hosted in the EU, and an open-weight model you run yourself. Open-weight means the trained model is published, so you can download it and run it on your own hardware.
The three options
Public API
You send a request over the internet to OpenAI, Anthropic or Google and get an answer back. You pay per token (a piece of a word) and run nothing yourself. You get the strongest models available, usually on the day they launch.
Your data leaves your environment and is processed by the provider. Business and API terms generally say it isn’t used to train their models, but it may be kept for a limited time, for example to detect abuse. You need a data processing agreement, and you should know where the processing happens.
Enterprise or EU-hosted API
The same or similar models through a cloud platform such as Microsoft Azure, AWS or Google Cloud with processing in EU regions, or from a European provider such as Mistral. You get an enterprise contract, a choice of region and often an account your IT team already manages. Data still leaves your servers, but it goes to a processor under contract, in a region you choose.
Self-hosted open-weight model
You run a model such as Llama, Mistral, Qwen or Gemma on your own servers, or on GPU machines in your own cloud account. Prompts and documents never leave your environment. In exchange, you are responsible for the hardware, the software, the updates and the security.
Three ways to use a language model
| Public API | EU-hosted API | Self-hosted | |
|---|---|---|---|
| Quality on hard tasks | The best available | The best or close to it | Good; the largest open models come close |
| Data stays in your environment | No | No | Yes |
| You choose where data is processed | Partly | Yes | Yes |
| Setup effort | Low | Low to medium | High |
| How you pay | Per token | Per token, or reserved capacity | Servers and people, mostly fixed |
| Works without internet | No | No | Yes |
| Who keeps it running | The provider | The provider | You or your supplier |
Privacy: what actually changes
The common fear is that an API provider will train its next model on your contracts. Under business and API terms that is generally not the case, but read the terms of the plan you actually buy. The real differences are elsewhere:
- Who processes the data. With an API, the provider is your processor under GDPR. You need a data processing agreement, a lawful basis for the processing and, if data leaves the EU, a legal basis for the transfer. Our guide to AI, GDPR and the EU AI Act covers this.
- Where it is processed. EU-hosted options settle most questions about where data lives. Some contracts and sector rules go further and forbid any outside processing at all.
- How long it is kept. Check how long prompts and outputs are stored, and whether you can shorten that.
Self-hosting is the right call when one of these is true:
- A contract, regulator or internal rule forbids sending the data to any third party. A law firm’s client files, a clinic’s patient notes or a bank’s internal reports are typical cases.
- The system must work offline, for example at a production site without reliable internet.
- You process a high, steady volume. More on that below.
Self-hosting isn’t automatically safe, though. You now own access control, patching and logging. And a model will tell anyone who asks whatever is in the documents it can read, so permissions still matter: the assistant should only search documents the person asking is allowed to see.
Quality: how big is the gap?
Open-weight models have improved quickly. For many everyday business tasks, such as pulling fields out of documents, sorting emails, summarising or answering questions from your own documents, a well-chosen mid-sized open model does the job.
The best API models are usually still ahead on long, multi-step reasoning, on agent work with many tools and on writing in smaller languages. If your users work in Slovak or Czech, test that specifically: smaller models are often noticeably weaker in them than in English.
Don’t decide on public benchmarks. Take 50 to 100 real examples from your process, run them through two or three candidate models and compare the results side by side with the people who do the work today.
Cost: it depends on volume
An API costs nothing when nobody uses it and grows in a straight line with use. A self-hosted model costs the same whether it is busy or idle. The GPU servers (machines with the graphics processors that run models) run around the clock, and someone has to look after them.
So self-hosting gets cheaper per request only when the hardware is kept busy. For a tool that 20 people use a few times a day, an API will almost always be cheaper. For a system that processes documents around the clock, the sums can tip the other way. Do them with your own numbers, and count people’s time, not just the server bill. What an AI project costs shows how to estimate the API side.
How much hardware? A rough rule: the memory a model needs is its number of parameters times the bytes stored per parameter. A model with 8 billion parameters at 16-bit precision needs about 16 GB just for its weights. Compressed to 4 bits (quantised, with a small loss of quality) it needs about 4–5 GB and runs on a single decent GPU. A 70-billion-parameter model needs about 140 GB at 16 bits, or around 35–40 GB at 4 bits: one or more data-centre GPUs. Every user working at the same time needs extra memory on top. The very largest open models need a whole server of data-centre GPUs, which few companies can justify.
Which option fits
- 1 Marketing drafts
- 2 Internal policy questions
- 3 Contract review at a law firm
- 4 Summaries of patient notes
- 5 Support reply drafts
- 6 Tagging a million product descriptions
What a self-hosted setup looks like
Running your own model takes more than downloading a file. A typical setup has these layers:
A self-hosted AI setup
- Your apps and users Chat, internal tools, automated workflows
- Gateway Login, permissions, logs, usage limits
- Retrieval Search over your documents, respecting who may see what
- Inference server Software that runs the model, such as vLLM or Ollama
- Open-weight model Llama, Mistral, Qwen, Gemma or similar
- GPU servers In your data centre or your cloud account
- Gateway. The single door to the model. It checks who is asking, logs requests and stops one runaway script from using all the capacity. It can also send some requests to an external API, which is how hybrid setups work.
- Retrieval. The model doesn’t know your documents. A search step finds the relevant passages and passes them to the model with the question. This is called RAG; see how RAG works.
- Inference server. The software that loads the model onto the GPUs and answers requests efficiently, for several users at once.
- Model and hardware. Chosen together. A model that doesn’t fit in GPU memory doesn’t run, or runs very slowly.
Then there is the work nobody sees: security updates, new model versions, monitoring speed and errors, and a plan for when a server fails.
Hybrid setups
You don’t have to pick one option for everything, just as you don’t have to build or buy everything. Common patterns:
- Route by sensitivity. Requests that involve personal or confidential data go to the self-hosted model, general tasks go to an API. The gateway decides by data source or by user group.
- Prove it on an API, then move. Run the pilot on an EU-hosted API with data you are allowed to use there. Move to self-hosting once you know the process works and what volume to expect.
- Small model in-house, big model for hard cases. A local model handles the routine steps. The rare hard case that contains no sensitive data goes to an API.
For example, a private clinic could use an API for appointment reminders and its public FAQ, and a self-hosted model to summarise patient notes for doctors. Same gateway, same login, two very different data rules.
Before you decide to self-host
Self-hosting readiness
- A written reason A contract, regulation, offline site or measured volume, not just a feeling
- Quality tested on your data Two or three open models on real examples, in your language
- Volume and peak load estimated Requests per day and people using it at the same time
- A hardware plan Buy, rent GPUs in your cloud, or use an EU provider that hosts open models
- An owner for operations Updates, security patches, monitoring, failures
- The model licence checked Some open models come with their own terms and usage restrictions
- Access rules defined Who may query which documents
- A fallback for when the server is down
Common mistakes
- Buying GPUs before testing models. Test first on rented hardware, then size what you buy.
- Comparing a server bill with an API bill. Self-hosting also costs people’s time. Count it.
- Assuming your own server means compliance. GDPR still applies: purpose, access, retention and records of processing.
- Picking the biggest model that fits. A smaller model is faster, cheaper and often good enough. Use a bigger one only where your tests show you need it.
What to do next
Start with the data, not the hardware. List what data the AI process needs, what your contracts and regulators say about it, and roughly how many requests you expect. In many cases an EU-hosted API is enough. Where it isn’t, a self-hosted model is a proven path.
We build private AI assistants that run in your own environment and connect to your documents and systems. If that’s what you need, see self-hosted LLMs and private AI.
