Overview

The LLM mode turns an Octelium Service into an identity-aware LLM gateway. It validates inference API requests, makes normalized LLM information available to Policies and dynamic configuration, injects upstream credentials and records inference-specific access logs. It also provides inference-aware plugins for prompts, tools, guardrails, model selection, reasoning, token rate limiting, semantic caching and semantic routing.

Here is a simple OpenAI gateway:

octeliumctl create secret openai-api-key

kind: Service
metadata:
  name: openai
spec:
  mode: LLM
  isPublic: true
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: openai-api-key

The isPublic field enables public clientless access to the Service (read more here). An authorized workload can use the Service URL as the base URL of an ordinary OpenAI client and use its Octelium access token instead of the provider API key.

Protocols

The protocol field describes the inference API spoken by downstream clients and, unless a translation is set, by the upstream as well. It defaults to OPENAI. Octelium validates and normalizes the following protocols:

OPENAI supports POST /v1/chat/completions, POST /v1/responses, POST /v1/completions, POST /v1/embeddings, POST /v1/moderations, GET /v1/models and GET /v1/models/{model}.
ANTHROPIC supports POST /v1/messages, POST /v1/messages/count_tokens, GET /v1/models and GET /v1/models/{model}.
GEMINI supports content generation, streaming content generation, token counting and single or batch embeddings over the Gemini Developer API /v1beta routes.
BEDROCK supports the Amazon Bedrock Runtime Converse and InvokeModel routes. Converse has normalized inference semantics. InvokeModel carries a model-native body, so it is proxied as raw inference without body parsing or inference-specific plugin processing.

Body-carrying operations require an application/json media type and a JSON object. Unsupported paths, methods and streaming combinations are rejected before reaching the upstream.

note

By default, the downstream and upstream of a Service speak the same protocol. The generation operation can be translated to an upstream that speaks another protocol as shown below. OpenAI-compatible servers such as vLLM and Ollama are supported when they expose the canonical routes. Azure OpenAI deployment routes and Vertex AI project-scoped routes are not the OPENAI and GEMINI protocols described here.

Operations and Routes

ctx.request.llm.operation describes what a request does independently of the provider protocol:

GENERATE generates output from an input.
EMBED generates vector embeddings.
MODERATE classifies an input for safety.
COUNT_TOKENS counts input tokens without generating output.
LIST_MODELS and GET_MODEL discover models.
RAW_INFERENCE represents a model-native Bedrock InvokeModel request whose body Octelium does not parse.

ctx.request.llm.route retains the exact API surface. Its values are CHAT_COMPLETIONS, RESPONSES, COMPLETIONS, EMBEDDINGS, MODERATIONS, MODELS_LIST, MODELS_GET, MESSAGES, COUNT_TOKENS, GENERATE_CONTENT, EMBED_CONTENT, CONVERSE and INVOKE_MODEL.

Use the operation in rules that should work across protocols and the route when the exact API shape matters.

Upstream Base Paths

The path in upstream.url acts as the provider base path. For OPENAI, ANTHROPIC and GEMINI, Octelium removes the leading version segment from the canonical downstream route and appends the remainder to that base path. The following example forwards /v1/chat/completions to /api/v1/chat/completions:

spec:
  mode: LLM
  config:
    upstream:
      url: https://openrouter.ai/api/v1
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: openrouter-api-key

When upstream.url has no path, the canonical downstream path is preserved. BEDROCK has no version prefix and appends its route as is. The path configuration or a path plugin can handle providers with another layout.

Protocol Translation

The translation field lets downstreams keep using whatever protocol and SDK they already use regardless of the protocol spoken by the upstream. The following example serves the OPENAI protocol to downstreams while the upstream is Anthropic:

octeliumctl create secret anthropic-api-key

kind: Service
metadata:
  name: claude
spec:
  mode: LLM
  config:
    upstream:
      url: https://api.anthropic.com
    llm:
      protocol: OPENAI
      translation:
        upstreamProtocol: ANTHROPIC
        defaultMaxOutputTokens: 4096
      auth:
        custom:
          header: x-api-key
          value:
            fromSecret: anthropic-api-key

protocol is the protocol served to downstreams and upstreamProtocol is the protocol spoken by the upstream. No translation is performed when upstreamProtocol is unset or equal to protocol. The OPENAI, ANTHROPIC, GEMINI and BEDROCK protocols can be translated to one another for the generation operation, in both streaming and non-streaming modes:

OPENAI translates the CHAT_COMPLETIONS route.
ANTHROPIC translates the MESSAGES route.
GEMINI translates the GENERATE_CONTENT route.
BEDROCK translates the CONVERSE route.

Other operations such as embeddings, moderations, token counting, model discovery and Bedrock InvokeModel have no counterpart in the other protocols and are rejected rather than proxied. The upstream base path rules described above apply to the upstream protocol route. The anthropic-version header is automatically set for an ANTHROPIC upstream when it is not explicitly set.

Octelium translates a bounded subset that the protocols have in common: text, images, system instructions, multi-turn history, client tools, tool calls and results, tool choice, stop sequences, output limits, sampling, streaming and usage. Provider-specific features that no other protocol defines (e.g. OpenAI structured outputs and logprobs, Anthropic thinking blocks, assistant prefills, citations and server tools, Gemini safety settings and built-in tools, and Bedrock guardrail configuration) are rejected instead of being silently dropped. Prompt cache controls that the target protocol does not define are dropped since they only affect cost. Tool schemas are proxied verbatim.

defaultMaxOutputTokens is used when the downstream protocol allows omitting the output limit while the upstream protocol requires one, which is the case of an ANTHROPIC upstream. It only applies to requests that declare no output limit. Zero uses a default of 4096. It is bounded by limits.maxOutputTokens, and a reasoning token budget decided by the Service is added on top of it. ctx.request.llm.maxOutputTokens still reports zero for such requests.

GEMINI and BEDROCK name the model in the request path. A translated request is therefore addressed to the model resolved by the Service (i.e. by model, a model plugin or a semantic router), or otherwise to the model requested by the downstream. A request that names no model or names a model that the upstream protocol cannot address is rejected.

The translation is applied after every inference-specific plugin on the request and before every one of them on the response. Policies, plugins, semantic caches and access logs therefore always see the downstream protocol. A downstream reasoning configuration is only translated when both protocols express reasoning in the same terms. Use the reasoning field (read more below) or a reasoning plugin to control reasoning for translated requests.

note

translation is global and only takes effect in the default configuration. Gemini tool-call signatures that must be returned on the next turn are retained by Octelium, bound to the Session and the upstream model, and restored into later translated requests.

Secretless Access

The auth field supports bearer tokens, custom API key headers, basic authentication, OAuth2 client credentials and AWS Signature Version 4 (read more here). The credential used to access the Octelium Service is independent from the provider credential. Downstream credentials are never forwarded to the provider.

Custom API Key

Anthropic uses an API key in x-api-key and requires a version header:

llm:
  protocol: ANTHROPIC
  auth:
    custom:
      header: x-api-key
      value:
        fromSecret: anthropic-api-key
  header:
    addRequestHeaders:
      - key: anthropic-version
        value: "2023-06-01"

Gemini can use the same authentication type with header: x-goog-api-key.

OAuth2 Client Credentials

llm:
  auth:
    oauth2ClientCredentials:
      clientID: llm-gateway
      clientSecret:
        fromSecret: provider-client-secret
      tokenURL: https://identity.example.com/oauth2/token
      scopes:
        - inference

Basic Authentication

llm:
  auth:
    basic:
      username: inference-client
      password:
        fromSecret: provider-password

AWS Signature Version 4

spec:
  mode: LLM
  config:
    upstream:
      url: https://bedrock-runtime.us-east-1.amazonaws.com
    llm:
      protocol: BEDROCK
      auth:
        sigv4:
          accessKeyID: AKIAEXAMPLE
          secretAccessKey:
            fromSecret: aws-secret-access-key
          region: us-east-1
          service: bedrock

Model Selection

By default, the model requested by the downstream is preserved. model can replace it with a static value, a CEL result or a Rego result.

Static Model

llm:
  model:
    value: gpt-5-mini

CEL

llm:
  model:
    eval: '"ai-admins" in ctx.user.spec.groups ? "gpt-5" : "gpt-5-mini"'

The CEL expression must return a string. An empty string preserves the downstream model.

Open Policy Agent

llm:
  model:
    opa: |
      package octelium.eval
      default result := "gpt-5-mini"
      result := "gpt-5" if {
        "ai-admins" in input.ctx.user.spec.groups
      }

The Rego script must produce a string through result. An empty result preserves the requested model. The normalized ctx.request.llm.model field always contains the downstream-requested value, even when the upstream model is replaced. Conditional model selection is also available as a plugin.

Reasoning

The reasoning field makes the Service, rather than the downstream, decide how much a reasoning model is allowed to think. It overwrites the downstream configuration for operations that support reasoning and is a no-op for other operations.

Portable Level

llm:
  reasoning:
    level: MEDIUM

The portable levels are NONE, MINIMAL, LOW, MEDIUM, HIGH, XHIGH and MAX. Octelium selects the strongest model-supported configuration that does not exceed the configured level. It never resolves upwards or silently drops an incompatible restriction.

Token Budget

llm:
  reasoning:
    tokenBudget: 4096

tokenBudget is exact and only works with models that accept numeric reasoning budgets. A model that only accepts ordinal effort is rejected. Zero is invalid; use level: NONE to disable reasoning.

Provider-Native Effort

llm:
  reasoning:
    effort: xhigh

effort is passed as an ordinal provider-native value. It is the escape hatch for an effort not represented by a portable level and is not translated to a numeric budget.

CEL

llm:
  reasoning:
    eval: 'ctx.request.llm.model == "gpt-5" ? "HIGH" : "NONE"'

The expression returns a portable level such as HIGH, a numeric budget such as 4096, or a provider-native effort. An empty result preserves the downstream reasoning configuration.

Open Policy Agent

llm:
  reasoning:
    opa: |
      package octelium.eval
      default result := "LOW"
      result := "MAX" if {
        "ai-admins" in input.ctx.user.spec.groups
      }

Conditional reasoning configuration is also available as a plugin. The last matching reasoning plugin overwrites this default.

Embeddings

Semantic caching and routing require an embedding configuration. Define it once in llm.embedding, or override it within an individual semantic plugin.

The current upstream can generate embeddings for OPENAI and GEMINI Services:

llm:
  embedding:
    source:
      currentUpstream: true
    model: text-embedding-3-small
    dimensions: 1024

Use a separate OPENAI or GEMINI embedding API when the main upstream does not provide compatible embeddings:

llm:
  embedding:
    source:
      upstream:
        url: https://embeddings.example.com/v1
        protocol: OPENAI
        auth:
          bearer:
            fromSecret: embeddings-api-key
    model: text-embedding-3-small

Zero dimensions uses the model default. Changing the model or dimensions makes vectors stored with the previous configuration unreachable. The embedding backend receives prompt-derived content after request guardrails have run.

Limits

llm:
  limits:
    maxRequestBytes: 8388608
    maxStreamEventBytes: 262144
    maxEstimatedInputTokens: 100000
    maxOutputTokens: 8192
    maxTools: 64
    maxToolSchemaBytes: 65536

maxRequestBytes bounds the JSON request body buffered and parsed by Octelium.
maxStreamEventBytes bounds inspection of an individual streamed event. A larger event is forwarded but not inspected.
maxEstimatedInputTokens rejects a request above Octelium's byte-based pre-flight estimate.
maxOutputTokens rejects a downstream-declared output maximum above the value. It does not insert a limit when the request declares none.
maxTools bounds the number of declared tools.
maxToolSchemaBytes bounds the serialized JSON Schema of one tool.

Zero uses an Octelium default or disables an optional inference limit, depending on the field. Internal hard limits always apply. The input estimate is not provider-accurate and must not be used for billing; read ctx.request.llm.estimateQuality alongside it.

Access Control

LLM-specific request information is stored in ctx.request.llm. The underlying HTTP request and parsed body are available in ctx.request.llm.http and ctx.request.llm.http.bodyMap (read more about HTTP access control here).

authorization:
  inlinePolicies:
    - spec:
        rules:
          - effect: ALLOW
            condition:
              all:
                of:
                  - match: ctx.request.llm.operation == "GENERATE"
                  - match: ctx.request.llm.route == "RESPONSES"
                  - match: ctx.request.llm.model == "gpt-5-mini"
                  - match: ctx.request.llm.stream == false
                  - match: ctx.request.llm.hasTools == false
                  - match: ctx.request.llm.hasImageInput == false
                  - match: ctx.request.llm.hasAudioInput == false
                  - match: ctx.request.llm.maxOutputTokens > 0
                  - match: ctx.request.llm.maxOutputTokens <= 4096
                  - match: '"ai-users" in ctx.user.spec.groups'

The normalized fields are:

protocol, operation, route, model and stream.
estimatedInputTokens, estimateQuality and maxOutputTokens.
hasTools, toolCount and the bounded toolNames list.
inputItemCount, hasImageInput and hasAudioInput.

The values describe the request sent by the downstream before model, prompt or reasoning plugins mutate it. Prefer them over inspecting an untrusted, deeply nested body whenever possible.

Dynamic Configuration

Dynamic configuration can select compatible upstreams, credentials and configuration from identity and normalized LLM context (read more here):

spec:
  mode: LLM
  config:
    upstream:
      url: https://default-provider.example.com/v1
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: default-provider-key
  dynamicConfig:
    configs:
      - name: premium
        upstream:
          url: https://premium-provider.example.com/v1
        llm:
          auth:
            bearer:
              fromSecret: premium-provider-key
          model:
            value: premium-model
    rules:
      - condition:
          all:
            of:
              - match: ctx.request.llm.model == "premium-model"
              - match: '"ai-admins" in ctx.user.spec.groups'
        configName: premium

protocol, translation and limits are global and only take effect in the default configuration. listenHTTP2 and cors are also global. Every selected upstream must speak the global upstream protocol, which is translation.upstreamProtocol when set and protocol otherwise.

HTTP Configuration

Since LLM is HTTP-based, config.llm also supports header and path manipulation, HTTP/2 and CORS (read more here):

llm:
  header:
    addRequestHeaders:
      - key: X-AI-Gateway
        value: octelium
      - key: X-User-ID
        eval: ctx.user.metadata.uid
    removeRequestHeaders:
      - X-Untrusted-Header
    addResponseHeaders:
      - key: X-Content-Source
        value: inference
    removeResponseHeaders:
      - Server
    forwardedMode: DROP
    authorizationMode: DELETE
    host:
      value: inference.internal.example.com
  path:
    addPrefix: /inference
  isUpstreamHTTP2: true
  listenHTTP2: true
  cors:
    allowOriginStringMatch:
      - https://ai-console.example.com
    allowMethods: POST, GET
    allowHeaders: Authorization, Content-Type
    allowCredentials: true
    allowClusterServices: true

Header and path changes apply after the downstream operation is validated. The host can instead be preserved or computed with eval, and forwardedMode can be DROP, OBFUSCATE or TRANSPARENT. isUpstreamHTTP2 affects connections to the provider. listenHTTP2 and cors are global because they affect downstream connections and unauthenticated preflight requests. Enabling cors.allowClusterServices trusts browser applications served by every Service in the Cluster.

Continue with the LLM plugins and LLM visibility references.