At some point in most AI projects, someone asks: “Are we even allowed to send this data to an AI company?” Then someone else asks whether you could just run your own model instead. Both are fair questions. The answer depends on your data, your volume and who will keep the thing running.
There are three common ways to use a large language model (LLM), the kind of model behind ChatGPT: a public API, an enterprise API hosted in the EU, and an open-weight model you run yourself. Open-weight means the trained model is published, so you can download it and run it on your own hardware.
The three options
Public API
You send a request over the internet to OpenAI, Anthropic or Google and get an answer back. You pay per token (a piece of a word) and run nothing yourself. The provider manages model serving and offers a catalogue of models; availability depends on the service and plan.
Your data leaves your environment and is processed by the provider. Check the terms for the specific service: whether inputs can be used for training, how long prompts and outputs are retained, and where processing happens. For personal data, assess the provider’s role and the necessary contractual safeguards.
Enterprise or EU-hosted API
A managed model service can offer regional processing, enterprise contracts and identity controls your IT team already uses. The deployment type matters: an EU storage location does not by itself establish where inference runs. Microsoft’s deployment documentation distinguishes global, data-zone and geography-based processing. AWS likewise distinguishes geographic and global inference routes. Check the exact model, route and contract before using company data.
Self-hosted open-weight model
You run a model such as Llama, Mistral, Qwen or Gemma on your own servers, or on GPU machines in your own cloud account. Prompts and documents can stay within that environment if the surrounding software, logging and integrations are configured to keep them there. Renting servers still involves a cloud provider. In exchange, you are responsible for the hardware, the software, the updates and the security.
Three ways to use a language model
| Public API | EU-hosted API | Self-hosted | |
|---|---|---|---|
| Quality on your tasks | Test available models | Test models in the chosen region | Test the model and hardware together |
| Data stays in your environment | No | No | Yes |
| Control over processing location | Depends on service | Depends on deployment and contract | Depends on your infrastructure |
| Setup effort | Low | Low to medium | High |
| How you pay | Per token | Per token, or reserved capacity | Servers and people, mostly fixed |
| Works without internet | No | No | If the whole setup is local |
| Who keeps it running | The provider | The provider | You or your supplier |
Privacy: what actually changes
Training use, retention and access are separate questions. A service can exclude model training while still keeping logs or using subprocessors. Read the terms of the service you actually plan to use:
- Who processes the data. For personal data, identify each party’s role and agree the required terms. The EDPB explains controller and processor responsibilities and the additional rules for transfers outside the EEA. Our guide to AI, GDPR and the EU AI Act covers this.
- Where it is processed. Check inference, storage, support access and subprocessors separately. Some contracts or internal policies restrict outside processing even when a service offers an EU region.
- How long it is kept. Check how long prompts and outputs are stored, and whether you can shorten that.
Self-hosting is worth evaluating when one of these applies:
- A specific contract or internal requirement calls for processing within infrastructure you control. Assess the actual requirement; the sector name alone does not decide which hosting model is permitted.
- The system must work offline, for example at a production site without reliable internet.
- You process a high, steady volume. More on that below.
Self-hosting isn’t automatically safe, though. You now own access control, patching and logging. A model may expose information from documents it receives, so permissions still matter: the assistant should only search documents the person asking is allowed to see.
Quality: how big is the gap?
Both API and self-hosted models can handle tasks such as extraction, classification and document questions. The useful comparison is whether a candidate meets your accuracy, language and response-time requirements on the hardware or service you would actually use.
There is no permanent quality ranking between these hosting options. For Slovak or Czech work, include those languages in the evaluation instead of relying on English benchmark scores.
Don’t decide on public benchmarks. Take 50 to 100 real examples from your process, run them through two or three candidate models and compare the results side by side with the people who do the work today.
Cost: it depends on volume
A usage-based API mainly charges for requests, though some plans also have subscriptions or reserved capacity. Self-hosting adds hardware or rental costs and operational work. An always-on server costs money while idle; a deployment that scales down can reduce that cost, with its own startup and capacity trade-offs.
Steady usage spreads fixed infrastructure costs across more requests. For an occasional tool, compare a usage-based API with the full cost of keeping a private deployment ready. For continuous processing, test throughput and price both options at the same quality level. Count people’s time as well as the server bill. What an AI project costs shows how to estimate the API side.
How much hardware? Weight size gives a starting point, not a full server specification. As an illustrative calculation, 8 billion parameters stored at 16 bits need about 16 GB for weights alone. Lower precision can reduce that footprint; Hugging Face’s quantization documentation explains the technique. Runtime overhead, context length and concurrent requests need additional memory. Test the exact model, precision and workload before buying hardware.
Which option fits
Data sensitivity →
- 1 Marketing drafts
- 2 Internal policy questions
- 3 Contract review at a law firm
- 4 Summaries of patient notes
- 5 Support reply drafts
- 6 Tagging a million product descriptions
Volume of requests →
- 1 Marketing drafts
- 2 Internal policy questions
- 3 Contract review at a law firm
- 4 Summaries of patient notes
- 5 Support reply drafts
- 6 Tagging a million product descriptions
What a self-hosted setup looks like
Running your own model takes more than downloading a file. A typical setup has these layers:
A self-hosted AI setup
- Your apps and users Chat, internal tools, automated workflows
- Gateway Login, permissions, logs, usage limits
- Retrieval Search over your documents, respecting who may see what
- Inference server Software that runs the model, such as vLLM or Ollama
- Open-weight model Llama, Mistral, Qwen, Gemma or similar
- GPU servers In your data centre or your cloud account
- Gateway. The single door to the model. It checks who is asking, logs requests and stops one runaway script from using all the capacity. It can also send some requests to an external API, which is how hybrid setups work.
- Retrieval. The model doesn’t know your documents. A search step finds the relevant passages and passes them to the model with the question. This is called RAG; see how RAG works.
- Inference server. The software that loads the model onto the GPUs and answers requests efficiently, for several users at once.
- Model and hardware. Chosen together. A model that doesn’t fit in GPU memory doesn’t run, or runs very slowly.
The running cost also includes security updates, new model versions, monitoring speed and errors, and a plan for when a server fails.
Hybrid setups
One entry point, different model routes
- Request arrives
- Check data rules Source, user and permitted processing
- Choose a model Approved API or private deployment
- Return the result Same application for the user
You don’t have to pick one option for everything, just as you don’t have to build or buy everything. Common patterns:
- Route by sensitivity. Requests that involve personal or confidential data go to the self-hosted model, general tasks go to an API. The gateway decides by data source or by user group.
- Prove it on an API, then move. Run the pilot on an EU-hosted API with data you are allowed to use there. Move to self-hosting once you know the process works and what volume to expect.
- Small model in-house, big model for hard cases. A local model handles the routine steps. The rare hard case that contains no sensitive data goes to an API.
For example, a clinic could route questions about its public opening hours to an approved API while keeping patient-note summaries in a private deployment. Appointment messages may contain personal data too, so their route needs the same explicit data assessment.
Before you decide to self-host
Self-hosting readiness
- A written reason A contract, regulation, offline site or measured volume, not just a feeling
- Quality tested on your data Two or three open models on real examples, in your language
- Volume and peak load estimated Requests per day and people using it at the same time
- A hardware plan Buy, rent GPUs in your cloud, or use an EU provider that hosts open models
- An owner for operations Updates, security patches, monitoring, failures
- The model licence checked Some open models come with their own terms and usage restrictions
- Access rules defined Who may query which documents
- A fallback for when the server is down
Common mistakes
- Buying GPUs before testing models. Test first on rented hardware, then size what you buy.
- Comparing a server bill with an API bill. Self-hosting also costs people’s time. Count it.
- Assuming your own server means compliance. GDPR still applies: purpose, access, retention and records of processing.
- Picking the biggest model that fits. A smaller model is faster, cheaper and often good enough. Use a bigger one only where your tests show you need it.
What to do next
Start with the data, not the hardware. List what data the AI process needs, what your contracts and regulators say about it, and roughly how many requests you expect. In many cases an EU-hosted API is enough. Where it isn’t, a self-hosted model is a proven path.
We build private AI assistants that run in your own environment and connect to your documents and systems. If that’s what you need, see self-hosted LLMs and private AI.
