The open-source, enterprise-ready AI gateway
Octelium is an open source, security-first, enterprise-ready AI gateway. Give every human and AI agent an identity, route each request dynamically across OpenAI, Anthropic, Gemini, Amazon Bedrock and compatible model servers, and understand exactly who uses which models, tokens and tools.
- Open source and self-hosted
- Four native provider protocols
- Horizontally scalable on Kubernetes
Humans and AI agents access OpenAI, Anthropic, Gemini and Amazon Bedrock through an Octelium AI gateway. Each inference request is authenticated, authorized from identity and context, routed to an effective model and audited with token and tool usage.
The caller determines what the request is allowed to become
Every person, application and AI agent authenticates as its own User. Octelium combines that identity with the normalized inference request to decide access, upstream destination, effective model, provider account and reasoning configuration.
- Policies evaluate protocol, operation, model, streaming, input modalities, declared tools and token limits alongside groups, session type, device posture and time.
- Model and reasoning can be selected with a static value, CEL expression or OPA policy. Reasoning supports portable levels, exact token budgets and provider-native effort values.
- Dynamic configuration can switch the entire upstream account, credential and LLM configuration, while semantic routing chooses a model from the meaning of the request.
kind: Policy
metadata:
name: production-inference
spec:
rules:
- effect: ALLOW
condition:
all:
of:
- match: ctx.request.llm.operation == "GENERATE"
- match: ctx.request.llm.model in ["auto", "gpt-5"]
- match: '"ai-platform" in ctx.user.spec.groups'
- match: ctx.session.status.type == "CLIENT"spec:
mode: LLM
config:
upstream:
url: https://api.openai.com
llm:
protocol: OPENAI
model:
eval: |
"ai-platform" in ctx.user.spec.groups
? "gpt-5" : "gpt-5-mini"
reasoning:
eval: |
ctx.user.spec.type == "WORKLOAD"
? "LOW" : "HIGH"llm:
plugins:
- name: route-by-intent
condition:
match: ctx.request.llm.model == "auto"
semanticRouter:
fallbackModel: gpt-5-mini
routes:
- name: complex-engineering
description: Architecture and debugging
examples:
- Diagnose this distributed deadlock
model: gpt-5
minSimilarity: 0.45Know who consumes every model, token and tool
Octelium produces structured, identity-aware LLM AccessLogs and an operational view built from them. Investigate one inference or rank usage across Users, Sessions, Services, Policies, models, operations, routes, providers and tools.
- Track requested, effective and provider-reported models, model rewrites, managed reasoning, finish reasons, response source and streaming event counts.
- Measure input, output, total, cached, cache-write and reasoning tokens with their provider-reported, estimated or partial usage state.
- See tools offered, called or removed, guardrail decisions, cache hits, semantic routes, quota outcomes, end-to-end latency and time to first token for each identity.
Selected request
Put safety, cost and quality controls in the request path
A deterministic post-authorization pipeline gives each control the right view of the request. Guardrails inspect original content, routing sees its semantic subject, caching keys the effective execution context, and a cache hit consumes no inference quota.
- Guardrails detect patterns, PII and secrets across prompts, system instructions, tool definitions and tool results. Deny, redact, strip or replace request content and inspect completed responses before release.
- Prompt controls own system instructions and messages, while tool controls allow, remove, deny, replace or add definitions and reconcile the requested tool choice.
- Token quotas reserve and reconcile usage per User, Session or tenant. Semantic caching partitions reusable answers; semantic routing sends each intent to an appropriate model.
- 01Guardrailsdetect · redact · deny
- 02Tool controlsallow · remove · replace
- 03Semantic routerintent → model
- 04Prompt controlsprepend · replace · reject
- 05Semantic cacheexact + vector match
- 06Token quotauser · session · tenant
One policy layer across the model APIs you already use
Octelium validates and normalizes each supported protocol without forcing applications through a lowest-common-denominator API. Clients keep the native wire format and streaming behavior of their provider.
- OpenAI support covers Responses, Chat Completions, Completions, Embeddings, Moderations and model discovery, including compatible servers exposing canonical routes.
- Anthropic Messages and token counting, Gemini generation and embeddings, and Amazon Bedrock Converse and InvokeModel are first-class protocols.
- Mistral, Meta AI, DeepSeek, Kimi, Grok, Qwen, Z.AI, Ollama, Hugging Face and other providers can sit behind the same controls when their endpoints expose the canonical OpenAI API routes.
Applications identify themselves without a shared provider key
Provider credentials stay inside the Cluster and are injected only after a request is authenticated and authorized. Each client presents its own Octelium identity, giving access a clear owner and audit trail without distributing upstream secrets.
- Upstream authentication supports bearer tokens, custom API-key headers, OAuth2 client credentials, basic authentication and AWS Signature Version 4.
- Downstream Authorization, X-Api-Key and Api-Key credentials are removed before proxying, preventing a caller credential from leaking to the provider.
- People authenticate through OIDC or SAML. Workloads use individual access tokens, OAuth2 client credentials or federated OIDC assertions.
kind: Service
metadata:
name: inference
spec:
mode: LLM
isPublic: true
config:
upstream:
url: https://api.openai.com
llm:
protocol: OPENAI
auth:
bearer:
fromSecret: openai-api-keykind: Service
metadata:
name: gemini
spec:
mode: LLM
config:
upstream:
url: https://generativelanguage.googleapis.com/v1beta
llm:
protocol: GEMINI
auth:
custom:
header: x-goog-api-key
value:
fromSecret: gemini-api-keyspec:
mode: LLM
config:
upstream:
url: https://bedrock-runtime.us-east-1.amazonaws.com
llm:
protocol: BEDROCK
auth:
sigv4:
accessKeyID: AKIAEXAMPLE
secretAccessKey:
fromSecret: aws-secret-access-key
region: us-east-1
service: bedrockOwn the gateway, the data plane and the audit stream
Octelium is free and open source and runs entirely on infrastructure you control. Production Clusters use Kubernetes to distribute and horizontally scale Gateway nodes, with no mandatory SaaS control plane in the request path.
- Deploy across managed or on-premises Kubernetes, place the data plane close to model endpoints, and scale Gateway replicas horizontally with demand.
- Export AccessLogs in real time over OpenTelemetry OTLP to your own observability, analytics and SIEM systems.
- Manage Services, Policies, identity providers, Secrets and plugins declaratively with GitOps-friendly YAML and programmable gRPC APIs.
- Seamlessly deploy, scale and serve containerized applications as Service upstreams.
$ octeliumctl apply ./ai-platform
Secret: model-provider-keys Updated
Policy: inference-access Created
Service: inference.ai Created
Service: embeddings.ai Created
Group: ai-platform Updated
IdentityProvider: workload-oidc Created
Cluster AI gateway configuration applied
4 resources created
2 resources updated
Gateway data plane reconciledThe controls behind an Octelium AI gateway
Each capability operates on the same authenticated and normalized request context, so access, routing, safety, efficiency and accountability do not drift into separate systems.
- 01
Identity and context
Human and workload identity, groups, session type, device posture and request attributes meet in one CEL or OPA authorization decision.
Read more - 02
Dynamic models and reasoning
A static value, CEL expression, Rego result or ordered plugin can choose the effective model and a portable reasoning level, exact token budget or provider-native effort.
Read more - 03
Prompt and tool governance
Prepend, append, replace, strip or reject instructions; inject messages; and allow, remove, deny, replace or add tool definitions.
Read more - 04
Guardrails
Inspect request content, instructions, tool definitions, tool results and completed responses using regex, PII detectors and secret detection.
Read more - 05
Token-based rate limiting
Reserve and reconcile input, output or total-token quotas over sliding windows, scoped per User, Session or an evaluated tenant key.
Read more - 06
Semantic caching
Reuse exact or semantically equivalent successful generations within a per-Session, per-User, shared or tenant-defined security partition.
Read more - 07
Semantic routing
Embed the request's semantic subject and select the highest-scoring model route, with a deterministic fallback when nothing matches.
Read more - 08
Extensible request path
Compose LLM-native controls with Lua, ExtProc, JSON Schema, direct responses, request rate limits and path plugins.
Read more
A generic reverse proxy sees an HTTP request. Octelium sees the identity, inference operation, model, reasoning, prompts, tools, token budget and the outcome of every control applied to them.
Frequently asked
- Which AI providers and APIs does Octelium support?
- The LLM mode natively understands OpenAI, Anthropic, Gemini and Amazon Bedrock protocols. It covers generation, streaming, embeddings, token counting, moderation and model discovery where the provider exposes those operations. OpenAI-compatible servers such as vLLM and Ollama work when they expose the canonical routes. Octelium validates and normalizes each protocol; it does not translate one provider's wire format into another.
- What makes the AI gateway identity-aware?
- Every request is tied to an Octelium human or workload User instead of relying on a shared provider key. Policies combine that identity, group membership, session type and device posture with normalized LLM attributes such as operation, requested model, streaming, modalities, tools and declared output tokens. The same context can choose an upstream account, model, reasoning level and plugin configuration dynamically.
- Can Octelium route requests to different models dynamically?
- Yes. Model and reasoning settings can be static, computed with CEL, returned by Open Policy Agent, or applied conditionally through plugins. Semantic routing classifies the meaning of a request against descriptions and examples and sends it to the best matching model. Dynamic Service configuration can additionally select a different upstream, provider account, credential, headers and complete LLM configuration.
- How does token-based rate limiting work?
- Token rate limiting applies a sliding-window quota to input, output or total inference tokens. Quotas can be keyed per User, per Session or by an evaluated value such as a tenant ID. Octelium reserves capacity before sending the request so concurrent requests observe one another, then reconciles the reservation with usage reported by the provider.
- What does semantic caching store and how is it isolated?
- The semantic cache can serve an exact repeat without an embedding call or find an earlier semantically equivalent request using vector similarity. The effective model, instructions, tools, reasoning, response schema and other execution context must still match. Cache partitions can be per Session, per User, shared or derived from context for tenant isolation. A cache hit uses no inference quota and still passes current response guardrails.
- Which guardrails and prompt controls are available?
- Guardrails inspect instructions, content, tool definitions, tool results and generated responses. They support RE2 patterns, built-in PII detectors and broad secret detection. Matching content can be denied, redacted, stripped or replaced where mutation is safe. Prompt plugins can prepend, append, replace, strip or reject system instructions and insert or modify user and assistant messages.
- What can teams see about AI usage?
- Identity-aware AccessLogs record who used which Service and model, requested and effective models, protocol, operation, routing decision, reasoning, finish reason, tools, guardrails, cache and quota outcomes, latency, time to first token and reported input, output, cached and reasoning token usage. The enterprise console aggregates these fields into trends, breakdowns and top users, models and tools.
- Is Octelium self-hosted and horizontally scalable?
- Yes. Octelium is free and open source, has no mandatory hosted control plane and runs on Kubernetes infrastructure you control. Production deployments distribute the data plane across Gateway nodes and scale it horizontally with Kubernetes. The same Cluster can deploy and scale containerized self-hosted inference servers as managed Service upstreams.
- Do applications receive the upstream provider credentials?
- No. Applications authenticate to Octelium with their own identity. Provider bearer tokens, custom API keys, OAuth2 client credentials, basic credentials and AWS SigV4 secrets stay in the Cluster and are injected only after authorization.
Related solutions
MCP gateway
An identity-aware Model Context Protocol gateway that authorizes JSON-RPC methods and individual tool calls.
Read moreZero trust infrastructure for AI agents
Identity, access control and auditing for AI agents, MCP servers and LLM providers at scale.
Read moreOpen source API gateway
Per-request authorization, secretless upstream credentials, rate limits and schema validation in front of your APIs.
Read moreDeploy Octelium on your own infrastructure in minutes
Free and open source. Self-hosted. No vendor lock-in.