LLM Gateway

The LLM mode turns an Octelium Service into an identity-aware LLM gateway. It validates supported inference API requests and makes normalized LLM information available for access control, dynamic configuration and visibility. It also enables secretless access to model providers and supports request and response manipulation using HTTP plugins.

Here is a simple OpenAI gateway:

octeliumctl create secret openai-api-key

kind: Service
metadata:
  name: openai
spec:
  mode: LLM
  isPublic: true
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: openai-api-key

The isPublic field enables public clientless access to the Service (read more here). An authorized workload can use the Service URL as the base URL of a standard OpenAI-compatible client and use its Octelium access token instead of the upstream provider API key.

Protocols and Operations

The protocol field describes the inference API exposed to downstream clients and expected from the upstream. It defaults to OPENAI when omitted. The following protocols are currently supported:

OPENAI supports POST /v1/chat/completions, POST /v1/responses, POST /v1/completions, POST /v1/embeddings, POST /v1/moderations, GET /v1/models and GET /v1/models/{model}.
ANTHROPIC supports POST /v1/messages, POST /v1/messages/count_tokens, GET /v1/models and GET /v1/models/{model}.

Body-carrying operations require an application/json media type and a JSON object. Streaming can be requested for OpenAI chat completions, responses and legacy completions, and for Anthropic messages. Unsupported paths, HTTP methods and streaming combinations are rejected before reaching the upstream.

note

The LLM mode validates and normalizes a protocol but does not translate between protocols. A Service using OPENAI must use an OpenAI-compatible upstream, while one using ANTHROPIC must use an Anthropic-compatible upstream. The OpenAI protocol also works with providers such as vLLM and Ollama that implement the same canonical /v1 routes. Provider APIs with different paths or wire formats, including the Azure OpenAI API, are not compatible with this mode unless an adapter is used upstream.

Here is an Anthropic example:

octeliumctl create secret anthropic-api-key

kind: Service
metadata:
  name: anthropic
spec:
  mode: LLM
  isPublic: true
  config:
    upstream:
      url: https://api.anthropic.com
    llm:
      protocol: ANTHROPIC
      auth:
        custom:
          header: x-api-key
          value:
            fromSecret: anthropic-api-key
      header:
        addRequestHeaders:
          - key: anthropic-version
            value: "2023-06-01"

Use the Anthropic API version required by your upstream. Request header manipulation is described in more detail here.

Secretless Access

The LLM mode supports bearer tokens, custom API key headers, basic authentication and OAuth2 client credentials for upstream access (read more about secretless access here). Before proxying an authorized request, Octelium removes downstream Authorization, X-Api-Key and Api-Key credentials and injects the configured upstream credential. This prevents a downstream credential from being accidentally forwarded to an LLM provider.

The upstream provider credential is independent from the credential used to access the Octelium Service. For clientless workloads, use an Octelium OAuth2 credential as illustrated here.

Model Selection

By default, Octelium forwards the model requested by the downstream. For request-body inference operations, you can override it with a static value as follows:

kind: Service
metadata:
  name: openai
spec:
  mode: LLM
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      model:
        value: gpt-5-mini
      auth:
        bearer:
          fromSecret: openai-api-key

The selected model can also be calculated with a CEL expression. For example, the following configuration selects a model using the identity of the User:

spec:
  mode: LLM
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      model:
        eval: '"ai-admins" in ctx.user.spec.groups ? "gpt-5" : "gpt-5-mini"'
      auth:
        bearer:
          fromSecret: openai-api-key

An empty result from model.eval preserves the model requested by the downstream. The request context and authorization rules always use the original downstream-requested model, even when the upstream request is rewritten to use another model. Model values generated by configuration are validated before the request is forwarded.

Limits

The LLM mode can enforce transport and inference-specific limits before a request reaches the provider:

spec:
  mode: LLM
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      limits:
        # Limit the entire request body to 8 MiB
        maxRequestBytes: 8388608
        # Limit an individual streamed event to 256 KiB
        maxStreamEventBytes: 262144
        # Apply a byte-based pre-flight estimate to text input
        maxEstimatedInputTokens: 100000
        # Reject a declared output limit above this value
        maxOutputTokens: 8192
        # Limit the number of declared tools
        maxTools: 64

A zero value uses Octelium's default or disables the optional inference-specific limit, depending on the field. Octelium always enforces internal upper bounds for the request body and streamed events.

maxEstimatedInputTokens uses Octelium's byte-based pre-flight estimate. It is neither provider-accurate nor an upper bound and must not be used for billing. The corresponding ctx.request.llm.estimateQuality value indicates whether the estimate was COMPLETE, PARTIAL or UNAVAILABLE. Non-text inputs such as images and audio make the estimate partial.

maxOutputTokens checks the maximum output value declared in a recognized request field. It does not impose an output token value when the downstream request does not declare one. Provider-side limits should therefore also be configured when a hard output ceiling is required.

Access Control

LLM-specific request information is stored in ctx.request.llm. The underlying HTTP request is available in ctx.request.llm.http, and the parsed JSON request body is available in ctx.request.llm.http.bodyMap when it is within the request-context body limit (read more about HTTP access control here).

Here is an example of an inline Policy that allows non-streamed chat requests for a particular model while rejecting tools and non-text input:

kind: Service
metadata:
  name: openai
spec:
  mode: LLM
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: openai-api-key
  authorization:
    inlinePolicies:
      - spec:
          rules:
            - effect: ALLOW
              condition:
                all:
                  of:
                    - match: ctx.request.llm.protocol == "OPENAI"
                    - match: ctx.request.llm.operation == "CHAT_COMPLETIONS"
                    - match: ctx.request.llm.model == "gpt-5-mini"
                    - match: ctx.request.llm.stream == false
                    - match: ctx.request.llm.hasTools == false
                    - match: ctx.request.llm.hasImageInput == false
                    - match: ctx.request.llm.hasAudioInput == false
                    - match: ctx.request.llm.maxOutputTokens <= 4096
                    - match: '"ai-users" in ctx.user.spec.groups'

The following normalized fields are available:

ctx.request.llm.protocol is OPENAI or ANTHROPIC.
ctx.request.llm.operation is an operation such as CHAT_COMPLETIONS, RESPONSES, EMBEDDINGS, MESSAGES or COUNT_TOKENS.
ctx.request.llm.model is the original model requested by the downstream.
ctx.request.llm.stream shows whether the downstream requested a streamed response.
ctx.request.llm.estimatedInputTokens and ctx.request.llm.estimateQuality contain the pre-flight estimate and its quality.
ctx.request.llm.maxOutputTokens is the output-token maximum declared by the downstream, or zero when none is declared.
ctx.request.llm.hasTools, ctx.request.llm.toolCount and ctx.request.llm.toolNames describe the tools declared by the request.
ctx.request.llm.inputItemCount is the number of messages, embedding inputs or other normalized input items.
ctx.request.llm.hasImageInput and ctx.request.llm.hasAudioInput describe non-text input modalities.

note

A zero ctx.request.llm.maxOutputTokens means that the downstream did not declare a recognized maximum. If a policy requires a downstream-supplied output limit, check that the value is greater than zero before comparing it to a maximum.

You can use ctx.request.llm.http.bodyMap when a policy needs a protocol field that is not normalized. Since request bodies can contain untrusted, deeply nested input, prefer the normalized fields whenever one is available.

Dynamic Configuration

You can route requests to different compatible providers, accounts or models based on identity and LLM request information (read more about dynamic configuration here). Here is an example that routes larger models to a separate OpenAI-compatible upstream:

kind: Service
metadata:
  name: llm
spec:
  mode: LLM
  config:
    upstream:
      url: https://small-models.example.com
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: small-models-api-key
  dynamicConfig:
    configs:
      - name: large-models
        upstream:
          url: https://large-models.example.com
        llm:
          auth:
            bearer:
              fromSecret: large-models-api-key
          model:
            value: large-production-model
    rules:
      - condition:
          all:
            of:
              - match: ctx.request.llm.model == "large-production-model"
              - match: '"ai-admins" in ctx.user.spec.groups'
        configName: large-models

The protocol and limits options are global validation settings and must be defined in the default Service configuration. Per-request configurations can change the upstream, upstream credential, selected model, headers, paths, plugins and visibility options. All dynamically selected upstreams must use the protocol configured by the default configuration because the gateway does not perform protocol translation.

Plugins

The LLM mode supports the same Lua and ExtProc plugins as the HTTP mode (read more here). Plugins can implement provider-specific validation, guardrails, redaction and intentional request or response mutation. A request body changed by a plugin is parsed again so that the updated normalized context is available to the processing stages that follow the plugin.

Here is a Lua plugin that is invoked for every chat-completion request:

spec:
  mode: LLM
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      plugins:
        - name: llm-guardrail
          condition:
            match: ctx.request.llm.operation == "CHAT_COMPLETIONS"
          lua:
            inline: |
              function onRequest(ctx)
                local body = json.decode(octelium.req.getRequestBody())
                -- Inspect, redact or replace fields in the request here.
                octelium.req.setRequestBody(json.encode(body))
              end

Plugins use the POST_AUTH phase by default, so authorization in that case is based on the request that existed before the plugin ran. A PRE_AUTH plugin can be defined in the default configuration when authorization must use the mutated request. A plugin must produce a valid request for the configured inference protocol and remain within the configured limits. The generic HTTP Cache plugin is not supported for the LLM mode because it does not understand model-provider cache behavior, authorization scope or semantic equivalence.

HTTP Configuration

Since LLM is an HTTP-based mode, it also supports HTTP header manipulation, path manipulation and HTTP/2 options under config.llm. These can be useful for OpenAI-compatible providers that expose the canonical operations below a path prefix:

spec:
  mode: LLM
  config:
    upstream:
      url: https://llm-provider.example.com
    llm:
      protocol: OPENAI
      path:
        addPrefix: /inference
      header:
        addRequestHeaders:
          - key: X-Gateway
            value: octelium
      upstreamHTTP2: true

Path and header changes apply to the upstream request after the downstream operation has been validated. You can read more about these HTTP options here.

Visibility

The Service emits identity-aware, LLM-specific access logs in real time to the audit collector. Each log includes the underlying HTTP information together with normalized fields such as protocol, operation, requested model, provider-returned model, streaming status, response identifier, finish reason and time to first token. When reported by the provider, the log can also include input, output, total, cached and reasoning token usage.

Provider-reported usage is marked as PROVIDER. Octelium can emit ESTIMATED or PARTIAL usage when complete provider usage is unavailable. These sources must be distinguished when using access logs for analytics or cost estimation.

Streamed responses produce a STREAM_START entry and a final STREAM_END entry. The final entry can include the stream duration, event count, time to first token, response metadata and accumulated usage. Non-streamed requests use the COMPLETE entry type.

Unlike the MCP mode, request and response body visibility is disabled by default for the LLM mode because prompts and model outputs commonly contain sensitive data. You can explicitly enable selected body and header fields as follows:

spec:
  mode: LLM
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      visibility:
        enableRequestBody: true
        enableRequestBodyMap: true
        enableResponseBody: true
        enableResponseBodyMap: true
        includeRequestHeaders:
          - User-Agent
        includeResponseHeaders:
          - Content-Type

Response body maps are available for non-streamed JSON responses. Body capture is bounded by Octelium and large bodies may be omitted from an AccessLog. Only enable body visibility when the audit backend and its retention policy are appropriate for prompts, uploaded content and generated output. Sensitive authentication and session headers are excluded from access logs even when all request or response headers are enabled.