Self-Hosted vs Managed LLM Gateway: How to Choose

TL;DR

• A self-hosted LLM gateway runs on your own infrastructure, so prompts and responses never pass through a third party; a managed LLM gateway is operated by a vendor and trades that control for zero operations.
• The self-hosted vs managed LLM gateway decision usually turns on data residency first, then cost at scale, then the team’s capacity to run another production service.
• For gateways that charge a percentage of spend, the break-even point is your fixed self-hosting cost divided by the fee rate; at a 5.5% fee and $2,000 a month in fixed costs, that is about $36,000 a month in model spend.
• A third model, a vendor-supported gateway inside your own VPC, keeps the data path private while shifting upgrades and uptime commitments to the vendor.
• Open-source gateways such as Bifrost can start as a single container and grow into a clustered, SLA-backed deployment without changing the API applications call.

The self-hosted vs managed LLM gateway question comes up as soon as more than one team starts calling model providers: someone has to decide whether the layer that routes, governs, and logs that traffic runs inside the company’s infrastructure or as a vendor’s service. Both options solve the same core problems (one API across providers, failover, spend tracking, access control), but they differ sharply in who sees the data, who carries the pager, and how cost scales. This guide compares the two on each of those dimensions, adds the in-VPC option most comparisons skip, and uses Bifrost, an open-source LLM gateway written in Go by Maxim AI, as a concrete reference for what self-hosting involves.

Self-Hosted vs Managed LLM Gateway: The Short Answer

Choose a self-hosted LLM gateway when prompts must stay inside your network, when model spend is high enough that a percentage fee costs more than an engineer’s time, or when you need governance the managed options do not offer. Choose a managed LLM gateway when the team is small, spend is modest, and nobody wants to operate another service.

Most teams sit somewhere in between, which is why the decision is better made on a few specific questions than on a general preference:

• Is any of this traffic subject to data residency, HIPAA, or contractual limits on third-party processors?
• What is the monthly model spend today, and what will it be in a year?
• Does the team already run production services on Kubernetes or a similar platform?
• Does the gateway need to govern MCP tool calls and agents, not only chat completions?

What Is a Self-Hosted LLM Gateway?

A self-hosted LLM gateway is gateway software that you deploy and run on infrastructure you control, such as a Kubernetes cluster, a VM, or a container service in your own cloud account. Applications send requests to it instead of directly to OpenAI, Anthropic, or other providers, and the gateway forwards them using provider keys stored in your environment.

Because the gateway sits in your network, the only external hop is the one to the model provider itself. Logs, cost data, and audit trails stay in your own database. The trade-off is that you own patching, scaling, monitoring, and backups.

Most self-hosted gateways are open source. A detailed comparison of the main options, including license, external dependencies, and air-gapped viability, is in this guide to the best open-source LLM gateways for self-hosted deployments.

What Is a Managed LLM Gateway?

A managed LLM gateway is a gateway the vendor operates as a service: you point your application at the vendor’s endpoint, configure providers and policies in a dashboard, and the vendor handles uptime, scaling, and upgrades. Setup typically takes minutes.

The cost of that convenience is that every prompt and response passes through the vendor’s infrastructure, and pricing is set by the vendor. Two common examples show how differently managed gateways charge:

A broader evaluation of managed and self-hosted gateways side by side, scored on overhead, governance, MCP support, and deployment model, is in this production-ready comparison of the top LLM gateways.

The Third Option: A Vendor-Supported Gateway in Your VPC

Most self-hosted vs managed comparisons present two choices. In practice, many enterprises pick a third: the gateway runs inside their own cloud account, so the data path stays private, while the vendor provides support, an uptime commitment, and a defined patch schedule.

Bifrost Enterprise is one example. Its in-VPC deployments run with no external network dependencies on GCP, AWS, Azure, Cloudflare, or Vercel infrastructure, and carry a 99.95% monthly uptime commitment for core components, 24/7 critical support, 14 days’ notice for major updates, and a choice between immediate or delayed security patches.

This model suits regulated teams that need the privacy of self-hosting but cannot staff round-the-clock operations for a gateway on their own.

Self-Hosted vs Managed LLM Gateway Compared

The table below compares the three deployment models on the dimensions that usually decide the question.

Cost: When Does Self-Hosting Pay Off?

In a self-hosted vs managed LLM gateway cost comparison, self-hosting pays off when the fixed cost of running the gateway is lower than what a managed gateway would charge for the same traffic. For managed gateways that charge a percentage of spend, the break-even point is simple to compute.

Break-even monthly model spend = fixed monthly self-hosting cost ÷ managed fee rate

The fixed cost has two parts: infrastructure (gateway instances, a database, load balancing) and the engineering time spent operating it. The figures below are illustrative assumptions, not quotes; replace them with your own.

Two points change the calculation in practice:

• A free managed gateway has no cost break-even. If the managed option charges nothing per request, as with Cloudflare’s core features, the decision rests entirely on data path, features, and governance.
• Engineering time is front-loaded. Initial setup, SSO, and alerting take most of the effort; a stable deployment needs far less time per month afterward, which pushes the break-even point lower over time.

Data Residency, Compliance, and the Data Path

Data residency is the most common deciding factor in a self-hosted vs managed LLM gateway evaluation. With a managed gateway, the vendor becomes a processor of every prompt and response, which means another entry in data processing agreements, vendor security reviews, and regional hosting questions.

With a self-hosted or in-VPC gateway, prompts travel only from your network to the model provider you already contract with. Logs, which often contain the most sensitive material, land in your own database or object storage. For teams that also run self-hosted models through vLLM or Ollama, a self-hosted gateway can keep that traffic entirely inside the network; the comparison of AI gateways for routing between vLLM, Ollama, and cloud LLMs covers how to keep restricted requests local while overflowing the rest to hosted APIs.

Governance follows the same logic. When the gateway runs in your environment, virtual keys, budgets, rate limits, and audit logs are enforced and stored under your control, which is the evidence auditors usually ask for. The Bifrost governance model applies those controls per team, per application, or per agent.

Latency and Reliability

A self-hosted gateway placed in the same region as your applications adds one internal network hop plus its own processing time. A managed gateway adds a round trip to the vendor’s nearest point of presence, which can be negligible or noticeable depending on where your applications and the vendor’s infrastructure sit.

Processing overhead varies widely between gateways. Bifrost’s published benchmarks report 11 microseconds of added latency per request at 5,000 requests per second on a t3.xlarge instance, with a 100% success rate, so in a self-hosted vs managed LLM gateway comparison the network path usually matters more than gateway processing time.

Reliability also shifts. A managed gateway’s uptime is the vendor’s responsibility, but its outage becomes your outage. A self-hosted gateway’s uptime is yours, which is why production deployments run multiple replicas: Bifrost Enterprise clustering recommends at least three nodes to tolerate one failure and supports rolling upgrades with mixed versions. Either way, automatic fallbacks between providers protect against the more frequent failure: a model provider returning errors.

What Self-Hosting Actually Involves

Self-hosting a gateway is the same work as running any stateful production service. Bifrost’s deployment requirements are a useful reference for what that means in practice:

• A container and a port. Bifrost ships as a Linux container for amd64 and arm64 and listens on port 8080, with a /health endpoint for readiness and liveness probes.
• A database. SQLite works for a single development instance; multiple replicas need PostgreSQL 16 or later.
• A stable encryption key. The key that protects stored provider credentials must stay the same across restarts, replicas, upgrades, and restores.
• Capacity planning. Bifrost’s enterprise sizing guidance calls for at least three gateway pods with 4 vCPU and 16 GB each, spread across availability zones, plus a PostgreSQL instance with a hot standby.
• A deployment path. The official Helm chart covers Kubernetes, and guides exist for EKS, GKE, AKS, ECS, Cloud Run, Fly.io, Render, Railway, and plain Docker.

Getting a first instance running takes one command, either npx -y @maximhq/bifrost or docker run -p 8080:8080 maximhq/bifrost, with no configuration file required. The production checklist above is where the real effort goes.

Supply chain responsibility

Self-hosting also means owning the gateway’s supply chain: which versions you pull, from where, and how quickly you patch. That risk became concrete on March 24, 2026, when two PyPI releases of LiteLLM (1.82.7 and 1.82.8) were published with credential-stealing code, an incident Datadog Security Labs linked to a wider campaign. LiteLLM’s official Docker image was not affected, and the malicious releases were removed quickly.

The lesson applies to any self-hosted gateway, not only Python ones:

• Pin exact image versions or digests instead of floating tags.
• Mirror images into a private registry and scan them before deployment.
• Prefer projects that publish their build and release controls. Bifrost documents its security practices, including dependency scanning, commit-pinned CI actions, and non-root container images.

A managed gateway moves this responsibility to the vendor, which is a real advantage for teams without a security function that reviews dependencies.

How to Choose a Self-Hosted vs Managed LLM Gateway

The decision table below maps common situations to the deployment model that usually fits. It assumes the gateway will carry production traffic, not only experiments.

Once the deployment model is settled, the next step is choosing the gateway itself. For self-hosted and in-VPC deployments, the open-source self-hosted LLM gateway comparison covers Kubernetes, air-gapped installation, and sizing for each option. For a mixed shortlist that includes managed services, the top LLM gateways comparison for production scores each gateway on overhead, governance depth, and deployment model. The LLM gateway buyer’s guide adds a capability matrix for procurement.

Where Bifrost Fits

Bifrost covers the self-hosted and in-VPC columns of the decision table with one codebase. The open-source gateway runs anywhere a container runs, exposes an OpenAI-compatible API as a drop-in replacement for existing SDKs, and includes failover, governance, and an MCP gateway. Bifrost Enterprise adds clustering, guardrails, SSO, audit logs, and the in-VPC option with an uptime commitment, so a team can start with a single container and move to an SLA-backed deployment without changing application code.

Beyond routing, Bifrost applies governance and security controls (virtual keys, budgets, guardrails, audit logs) centrally, and Bifrost Edge extends the same governance and security to AI traffic on employee machines, with endpoint enforcement on each device. Bifrost Edge is currently in alpha.

For more on the open-source landscape, this site also covers open-source LLM gateways worth running in production and LiteLLM alternatives.

Frequently Asked Questions

What is the difference between a self-hosted and a managed LLM gateway?

A self-hosted LLM gateway runs on infrastructure you control, so prompts, logs, and keys stay in your environment and your team operates it. A managed LLM gateway is run by a vendor as a service, so setup is faster and there is nothing to operate, but every request passes through the vendor’s infrastructure and pricing is set by the vendor.

Is a self-hosted LLM gateway cheaper than a managed one?

A self-hosted LLM gateway is cheaper once its fixed cost (infrastructure plus engineering time) falls below what a managed gateway charges for the same traffic. Against a percentage fee, divide the fixed monthly cost by the fee rate to find the break-even spend. Against a free managed gateway, self-hosting is not cheaper, so the choice rests on data control and features.

When should you self-host an AI gateway?

Self-host an AI gateway when prompts must stay inside your network for regulatory or contractual reasons, when model spend is high enough that a percentage fee exceeds operating costs, when you need air-gapped deployment, or when you need governance over agents and MCP tools that managed options do not provide. Teams with an existing Kubernetes platform face the lowest added cost.

Does a self-hosted gateway add latency?

A self-hosted gateway adds one internal network hop plus its processing overhead, which is small for gateways written in compiled languages. Bifrost, for example, adds about 11 microseconds per request at 5,000 requests per second in its published benchmark. A managed gateway adds a round trip to the vendor’s infrastructure, which depends on region.

Can you self-host Bifrost?

Yes. Bifrost is open source under the Apache 2.0 license and runs as a single container with npx or Docker, or on Kubernetes through the official Helm chart. Deployment guides cover EKS, GKE, AKS, ECS, Cloud Run, and other platforms. Bifrost Enterprise adds clustering, guardrails, and an in-VPC deployment option with a 99.95% uptime commitment.

What is a hybrid or in-VPC LLM gateway deployment?

A hybrid or in-VPC LLM gateway deployment runs the gateway inside your own cloud account, so the data path stays private, while the vendor supplies support, patches, and an uptime commitment. It combines the data control of self-hosting with part of the operational relief of a managed service, and suits regulated teams without dedicated 24/7 operations.

Next Steps

The self-hosted vs managed LLM gateway choice is mostly a question of where data may flow and who will run the service, with cost acting as the tiebreaker at scale.