---
title: "Open Source AI Gateway for Secure Enterprise AI - Octelium"
description: "Open source, self-hosted AI gateway for OpenAI, Anthropic, Gemini and Amazon Bedrock with identity-aware access control, protocol translation, guardrails, semantic routing, caching, token controls and audit logs."
canonical: "https://octelium.com/solutions/open-source-ai-gateway"
---

Open-source AI gateway

# Secure, govern and observe LLM access

**Octelium is a free and open source, self-hosted AI gateway for controlled access to model providers and self-hosted inference.** Authenticate every human, application and AI agent, authorize each inference request with identity-aware policy, keep provider credentials out of applications, enforce prompt, tool and token controls, and record structured LLM audit data.

[Install in minutes](https://octelium.com/docs/octelium/latest/overview/quick-install.md)[Live Demo](https://demo.octelium.com)[GitHub repo](https://github.com/octelium/octelium)

- Open source
- Self-hosted
- Kubernetes-native
- OpenTelemetry-ready

Identity-aware access control

## Control who can use each model and what they can ask it to do

Every request is associated with an authenticated Octelium User. Authorization evaluates that identity together with normalized inference context before the request reaches the provider.

- [Apply CEL or OPA policy to human and workload identity, groups, session type, device posture and request attributes together with protocol, operation, model, streaming mode, tools and token limits.](https://octelium.com/docs/octelium/latest/management/core/policy.md)
- [Allow or deny access and constrain the effective model or reasoning configuration from the same request context used for authorization.](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#model-selection)
- [Select upstreams, provider accounts, credentials and LLM configuration dynamically; use semantic routing when model choice should depend on request intent.](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-router)

[Read about LLM access control](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md)

AI security and governance

## Apply LLM-aware controls to every inference request

Octelium understands the semantics of supported LLM protocols instead of treating inference as opaque HTTP. Inference-aware plugins provide a common control layer for safety, model selection, usage limits and optimization.

- [Apply guardrails to prompts, tool definitions, tool results and model responses, with controls for sensitive data, secrets and configured content patterns.](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#guardrail)
- [Govern system prompts and tool definitions before they reach the model, including adding, replacing, filtering or rejecting application-supplied instructions and tools.](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#prompt)
- [Set token-based rate limits by identity, session or tenant so shared provider accounts do not become shared, uncontrolled inference budgets.](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#token-rate-limit)
- [Use semantic routing to choose models by request intent and semantic caching to reuse compatible responses within explicit identity or tenant boundaries.](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-router)

[Explore LLM plugins and controls](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md)

Provider credential isolation

## Keep model-provider credentials out of applications and agents

Applications authenticate to Octelium with their own identity instead of receiving a shared provider key. After authorization, the gateway supplies the credential required by the selected upstream and prevents downstream authentication material from being forwarded to the provider.

- [Authenticate upstreams with bearer tokens, custom API-key headers, OAuth2 client credentials, basic authentication or AWS Signature Version 4.](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#secretless-access)
- [Store provider credentials centrally as Octelium Secrets and inject them only after the request has been authenticated and authorized.](https://octelium.com/docs/octelium/latest/management/core/service/secretless.md)
- [Authenticate people through OIDC or SAML and workloads through individual access tokens, OAuth2 client credentials or federated OIDC assertions.](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md)

[Read about secretless access](https://octelium.com/docs/octelium/latest/management/core/service/secretless.md)

Multi-provider AI gateway

## Connect applications to OpenAI, Anthropic, Gemini and Bedrock through one gateway

Octelium natively understands all four supported inference protocols and can translate between them in any direction. Applications can keep the API format that fits them while the gateway selects a different provider or protocol upstream.

- [Translate any supported downstream protocol to any supported upstream protocol across OpenAI, Anthropic, Gemini and Amazon Bedrock.](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#protocols)
- [Switch providers, models or provider accounts centrally without requiring every application or agent to implement each provider API independently.](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#model-selection)
- [Normalize provider-specific requests into common LLM operations and context so access policy, guardrails, routing, token controls and observability remain consistent across providers.](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#operations-and-routes)
- [Use OpenAI-compatible model servers alongside the native OpenAI, Anthropic, Gemini and Bedrock integrations for hosted or self-hosted inference.](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#protocols)

[See supported providers and protocols](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#protocols)

AI observability and audit

## Trace AI usage by identity, model, policy, tool and token

Every LLM Service emits structured, inference-specific AccessLogs linked to the authenticated User, Session, Service and authorization decision. Investigate individual requests or aggregate usage across identities, applications, models and policies.

- [Record requested, effective and provider-reported models, including model rewrites and semantic-routing decisions, together with managed reasoning configuration and finish reasons.](https://octelium.com/docs/octelium/latest/management/core/service/llm/visibility.md)
- [Measure provider-reported input, output, total, cached and reasoning-token usage, while distinguishing complete, partial and estimated measurements.](https://octelium.com/docs/octelium/latest/management/core/service/llm/visibility.md#provider-token-usage)
- [Record tools offered, called or removed by policy, guardrail outcomes, cache and quota decisions, latency, streaming behavior and time to first token.](https://octelium.com/docs/octelium/latest/management/core/service/llm/visibility.md)
- [Prompt and completion body capture is disabled by default; enable only the fields required for operational or security analysis.](https://octelium.com/docs/octelium/latest/management/core/service/llm/visibility.md#body-and-header-visibility)

[See LLM visibility fields](https://octelium.com/docs/octelium/latest/management/core/service/llm/visibility.md)

Routing, caching and usage control

## Route models intelligently and control inference consumption

LLM-native plugins let platform teams optimize model usage without moving those decisions into every application. Routing, caching and token controls can use the same identity and request context as access policy.

- [Select models dynamically from identity and request context, including semantic routing that chooses an appropriate model from the meaning of the request.](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-router)
- [Use semantic caching to reuse equivalent successful responses and reduce repeated provider calls while preserving explicit User, Session or tenant isolation.](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-cache)
- [Apply token-based rate limits to Users, Sessions, workloads or tenant-defined keys to control shared inference capacity and prevent unbounded consumption.](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#token-rate-limit)
- [Control model and reasoning configuration centrally with static policy, CEL, OPA or conditional plugins rather than duplicating provider logic across applications.](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#model-selection)

[Explore routing, caching and token controls](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md)

Self-hosted deployment

## Run the AI gateway on infrastructure you control

Octelium runs on Kubernetes and does not require a hosted control plane in the inference request path. Keep the gateway, provider credentials and audit stream inside your own cloud or on-premises environment.

- [Deploy Gateway nodes across managed or self-hosted Kubernetes environments and scale the data plane horizontally with demand.](https://octelium.com/docs/octelium/latest/overview/how-octelium-works.md)
- [Export AccessLogs in real time over OpenTelemetry OTLP to existing observability, analytics and SIEM systems.](https://octelium.com/docs/octelium/latest/management/core/visibility.md)
- [Manage the entire Cluster declaratively with YAML and GitOps, or program it directly through gRPC APIs and the Go, TypeScript, Rust and Python SDKs.](https://octelium.com/docs/octelium/latest/overview/management.md)
- [Deploy, scale and serve containerized applications directly as Service upstreams.](https://octelium.com/docs/octelium/latest/management/core/service/managed-containers.md)

[Read the management guide](https://octelium.com/docs/octelium/latest/overview/management.md)

AI gateway capabilities

## Security and control across the inference path

Authorization, provider authentication, routing, inference controls and audit records operate on the same authenticated and normalized request context.

1. 01

   ### Identity and context

   Human and workload identity, groups, session type, device posture and request attributes meet in one CEL or OPA authorization decision.

   [Read more](https://octelium.com/docs/octelium/latest/management/core/policy.md)

2. 02

   ### Dynamic models and reasoning

   A static value, CEL expression, Rego result or ordered plugin can choose the effective model and a portable reasoning level, exact token budget or provider-native effort.

   [Read more](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#model-selection)

3. 03

   ### Prompt and tool governance

   Prepend, append, replace, strip or reject instructions; inject messages; and allow, remove, deny, replace or add tool definitions.

   [Read more](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#prompt)

4. 04

   ### Guardrails

   Inspect request content, instructions, tool definitions, tool results and completed responses using regex, PII detectors and secret detection.

   [Read more](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#guardrail)

5. 05

   ### Token-based rate limiting

   Reserve and reconcile input, output or total-token quotas over sliding windows, scoped per User, Session or an evaluated tenant key.

   [Read more](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#token-rate-limit)

6. 06

   ### Semantic caching

   Reuse exact or semantically equivalent successful generations within a per-Session, per-User, shared or tenant-defined security partition.

   [Read more](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-cache)

7. 07

   ### Semantic routing

   Embed the request's semantic subject and select the highest-scoring model route, with a deterministic fallback when nothing matches.

   [Read more](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-router)

8. 08

   ### Extensible request path

   Compose LLM-native controls with Lua, ExtProc, JSON Schema, direct responses, request rate limits and path plugins.

   [Read more](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#http-plugins)

Questions

## Frequently asked

- Which AI providers and APIs does Octelium support?

  The LLM mode natively understands OpenAI, Anthropic, Gemini and Amazon Bedrock protocols and can translate between any of the four in either direction. It covers generation, streaming, embeddings, token counting, moderation and model discovery where the selected protocol exposes those operations. OpenAI-compatible servers such as vLLM and Ollama can also be used when they expose the supported OpenAI API surface.

- What makes the AI gateway identity-aware?

  Every request is tied to an Octelium human or workload User instead of relying on a shared provider key. Policies combine that identity, group membership, session type and device posture with normalized LLM attributes such as operation, requested model, streaming, modalities, tools and declared output tokens. The same context can choose an upstream account, model, reasoning level and plugin configuration dynamically.

- Can Octelium route requests to different models dynamically?

  Yes. Model and reasoning settings can be static, computed with CEL, returned by Open Policy Agent, or applied conditionally through plugins. Semantic routing classifies the meaning of a request against descriptions and examples and sends it to the best matching model. Dynamic Service configuration can additionally select a different upstream, provider account, credential, headers and complete LLM configuration.

- How does token-based rate limiting work?

  Token rate limiting applies a sliding-window quota to input, output or total inference tokens. Quotas can be keyed per User, per Session or by an evaluated value such as a tenant ID. Octelium reserves capacity before sending the request so concurrent requests observe one another, then reconciles the reservation with usage reported by the provider.

- What does semantic caching store and how is it isolated?

  The semantic cache can serve an exact repeat without an embedding call or find an earlier semantically equivalent request using vector similarity. The effective model, instructions, tools, reasoning, response schema and other execution context must still match. Cache partitions can be per Session, per User, shared or derived from context for tenant isolation. A cache hit uses no inference quota and still passes current response guardrails.

- Which guardrails and prompt controls are available?

  Guardrails inspect instructions, content, tool definitions, tool results and generated responses. They support RE2 patterns, built-in PII detectors and broad secret detection. Matching content can be denied, redacted, stripped or replaced where mutation is safe. Prompt plugins can prepend, append, replace, strip or reject system instructions and insert or modify user and assistant messages.

- What can teams see about AI usage?

  Identity-aware AccessLogs record who used which Service and model, requested and effective models, protocol, operation, routing decision, reasoning, finish reason, tools, guardrails, cache and quota outcomes, latency, time to first token and reported input, output, cached and reasoning token usage. The enterprise console aggregates these fields into trends, breakdowns and top users, models and tools.

- Is Octelium self-hosted and horizontally scalable?

  Yes. Octelium is free and open source, has no mandatory hosted control plane and runs on Kubernetes infrastructure you control. Production deployments distribute the data plane across Gateway nodes and scale it horizontally with Kubernetes. The same Cluster can deploy and scale containerized self-hosted inference servers as managed Service upstreams.

- Do applications receive the upstream provider credentials?

  No. Applications authenticate to Octelium with their own identity. Provider bearer tokens, custom API keys, OAuth2 client credentials, basic credentials and AWS SigV4 secrets stay in the Cluster and are injected only after authorization.

Keep reading

## Related solutions

### [MCP gateway](https://octelium.com/solutions/open-source-mcp-gateway.md)

[A self-hosted, identity-aware MCP gateway with tool-level access control, guardrails, credential isolation and protocol-specific auditing.](https://octelium.com/solutions/open-source-mcp-gateway.md)

[Read more](https://octelium.com/solutions/open-source-mcp-gateway.md)

### [Secure infrastructure for AI agents](https://octelium.com/solutions/agentic-ai-secure-access-zero-trust.md)

[Identity-aware access control, credential isolation and auditing for AI agents accessing MCP servers, LLM providers and infrastructure.](https://octelium.com/solutions/agentic-ai-secure-access-zero-trust.md)

[Read more](https://octelium.com/solutions/agentic-ai-secure-access-zero-trust.md)

### [Open source API gateway](https://octelium.com/solutions/open-source-api-gateway.md)

[Identity-aware API access with per-request Policy, upstream credential isolation, rate limiting, validation and request processing.](https://octelium.com/solutions/open-source-api-gateway.md)

[Read more](https://octelium.com/solutions/open-source-api-gateway.md)

Get started

## Run Octelium on your own infrastructure

Install a self-hosted Cluster and start applying identity-aware access control to your Services.

[Quick install guide](https://octelium.com/docs/octelium/latest/overview/quick-install.md)[GitHub repo](https://github.com/octelium/octelium)[Enterprise package](https://octelium.com/enterprise.md)
