Overview

The LLM mode turns an Octelium Service into an identity-aware LLM gateway. It validates inference API requests, makes normalized LLM information available to Policies and dynamic configuration, injects upstream credentials and records inference-specific access logs. It also provides inference-aware plugins for prompts, tools, guardrails, model selection, reasoning, token rate limiting, semantic caching and semantic routing.

Here is a simple OpenAI gateway:

octeliumctl create secret openai-api-key

kind: Service
metadata:
  name: openai
spec:
  mode: LLM
  isPublic: true
  config:
    upstream:
      url: https://api.openai.com
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: openai-api-key

The isPublic field enables public clientless access to the Service (read more here). An authorized workload can use the Service URL as the base URL of an ordinary OpenAI client and use its Octelium access token instead of the provider API key.

Protocols

The protocol field describes the inference API spoken by both downstream clients and the upstream. It defaults to OPENAI. Octelium validates and normalizes the following protocols:

OPENAI supports POST /v1/chat/completions, POST /v1/responses, POST /v1/completions, POST /v1/embeddings, POST /v1/moderations, GET /v1/models and GET /v1/models/{model}.
ANTHROPIC supports POST /v1/messages, POST /v1/messages/count_tokens, GET /v1/models and GET /v1/models/{model}.
GEMINI supports content generation, streaming content generation, token counting and single or batch embeddings over the Gemini Developer API /v1beta routes.
BEDROCK supports the Amazon Bedrock Runtime Converse and InvokeModel routes. Converse has normalized inference semantics. InvokeModel carries a model-native body, so it is proxied as raw inference without body parsing or inference-specific plugin processing.

Body-carrying operations require an application/json media type and a JSON object. Unsupported paths, methods and streaming combinations are rejected before reaching the upstream.

note

The LLM mode does not translate between protocols. The downstream and upstream of a Service must speak the same protocol. OpenAI-compatible servers such as vLLM and Ollama are supported when they expose the canonical routes. Azure OpenAI deployment routes and Vertex AI project-scoped routes are not the OPENAI and GEMINI protocols described here.

Operations and Routes

ctx.request.llm.operation describes what a request does independently of the provider protocol:

GENERATE generates output from an input.
EMBED generates vector embeddings.
MODERATE classifies an input for safety.
COUNT_TOKENS counts input tokens without generating output.
LIST_MODELS and GET_MODEL discover models.
RAW_INFERENCE represents a model-native Bedrock InvokeModel request whose body Octelium does not parse.

ctx.request.llm.route retains the exact API surface. Its values are CHAT_COMPLETIONS, RESPONSES, COMPLETIONS, EMBEDDINGS, MODERATIONS, MODELS_LIST, MODELS_GET, MESSAGES, COUNT_TOKENS, GENERATE_CONTENT, EMBED_CONTENT, CONVERSE and INVOKE_MODEL.

Use the operation in rules that should work across protocols and the route when the exact API shape matters.

Upstream Base Paths

The path in upstream.url acts as the provider base path. For OPENAI, ANTHROPIC and GEMINI, Octelium removes the leading version segment from the canonical downstream route and appends the remainder to that base path. The following example forwards /v1/chat/completions to /api/v1/chat/completions:

spec:
  mode: LLM
  config:
    upstream:
      url: https://openrouter.ai/api/v1
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: openrouter-api-key

When upstream.url has no path, the canonical downstream path is preserved. BEDROCK has no version prefix and appends its route as is. The path configuration or a path plugin can handle providers with another layout.

Secretless Access

The auth field supports bearer tokens, custom API key headers, basic authentication, OAuth2 client credentials and AWS Signature Version 4 (read more here). The credential used to access the Octelium Service is independent from the provider credential. Downstream credentials are never forwarded to the provider.

Custom API Key

Anthropic uses an API key in x-api-key and requires a version header:

llm:
  protocol: ANTHROPIC
  auth:
    custom:
      header: x-api-key
      value:
        fromSecret: anthropic-api-key
  header:
    addRequestHeaders:
      - key: anthropic-version
        value: "2023-06-01"

Gemini can use the same authentication type with header: x-goog-api-key.

OAuth2 Client Credentials

llm:
  auth:
    oauth2ClientCredentials:
      clientID: llm-gateway
      clientSecret:
        fromSecret: provider-client-secret
      tokenURL: https://identity.example.com/oauth2/token
      scopes:
        - inference

Basic Authentication

llm:
  auth:
    basic:
      username: inference-client
      password:
        fromSecret: provider-password

AWS Signature Version 4

spec:
  mode: LLM
  config:
    upstream:
      url: https://bedrock-runtime.us-east-1.amazonaws.com
    llm:
      protocol: BEDROCK
      auth:
        sigv4:
          accessKeyID: AKIAEXAMPLE
          secretAccessKey:
            fromSecret: aws-secret-access-key
          region: us-east-1
          service: bedrock

Model Selection

By default, the model requested by the downstream is preserved. model can replace it with a static value, a CEL result or a Rego result.

Static Model

llm:
  model:
    value: gpt-5-mini

CEL

llm:
  model:
    eval: '"ai-admins" in ctx.user.spec.groups ? "gpt-5" : "gpt-5-mini"'

The CEL expression must return a string. An empty string preserves the downstream model.

Open Policy Agent

llm:
  model:
    opa: |
      package octelium.eval
      default result := "gpt-5-mini"
      result := "gpt-5" if {
        "ai-admins" in input.ctx.user.spec.groups
      }

The Rego script must produce a string through result. An empty result preserves the requested model. The normalized ctx.request.llm.model field always contains the downstream-requested value, even when the upstream model is replaced. Conditional model selection is also available as a plugin.

Reasoning

The reasoning field makes the Service, rather than the downstream, decide how much a reasoning model is allowed to think. It overwrites the downstream configuration for operations that support reasoning and is a no-op for other operations.

Portable Level

llm:
  reasoning:
    level: MEDIUM

The portable levels are NONE, MINIMAL, LOW, MEDIUM, HIGH, XHIGH and MAX. Octelium selects the strongest model-supported configuration that does not exceed the configured level. It never resolves upwards or silently drops an incompatible restriction.

Token Budget

llm:
  reasoning:
    tokenBudget: 4096

tokenBudget is exact and only works with models that accept numeric reasoning budgets. A model that only accepts ordinal effort is rejected. Zero is invalid; use level: NONE to disable reasoning.

Provider-Native Effort

llm:
  reasoning:
    effort: xhigh

effort is passed as an ordinal provider-native value. It is the escape hatch for an effort not represented by a portable level and is not translated to a numeric budget.

CEL

llm:
  reasoning:
    eval: 'ctx.request.llm.model == "gpt-5" ? "HIGH" : "NONE"'

The expression returns a portable level such as HIGH, a numeric budget such as 4096, or a provider-native effort. An empty result preserves the downstream reasoning configuration.

Open Policy Agent

llm:
  reasoning:
    opa: |
      package octelium.eval
      default result := "LOW"
      result := "MAX" if {
        "ai-admins" in input.ctx.user.spec.groups
      }

Conditional reasoning configuration is also available as a plugin. The last matching reasoning plugin overwrites this default.

Embeddings

Semantic caching and routing require an embedding configuration. Define it once in llm.embedding, or override it within an individual semantic plugin.

The current upstream can generate embeddings for OPENAI and GEMINI Services:

llm:
  embedding:
    source:
      currentUpstream: true
    model: text-embedding-3-small
    dimensions: 1024

Use a separate OPENAI or GEMINI embedding API when the main upstream does not provide compatible embeddings:

llm:
  embedding:
    source:
      upstream:
        url: https://embeddings.example.com/v1
        protocol: OPENAI
        auth:
          bearer:
            fromSecret: embeddings-api-key
    model: text-embedding-3-small

Zero dimensions uses the model default. Changing the model or dimensions makes vectors stored with the previous configuration unreachable. The embedding backend receives prompt-derived content after request guardrails have run.

Limits

llm:
  limits:
    maxRequestBytes: 8388608
    maxStreamEventBytes: 262144
    maxEstimatedInputTokens: 100000
    maxOutputTokens: 8192
    maxTools: 64
    maxToolSchemaBytes: 65536

maxRequestBytes bounds the JSON request body buffered and parsed by Octelium.
maxStreamEventBytes bounds inspection of an individual streamed event. A larger event is forwarded but not inspected.
maxEstimatedInputTokens rejects a request above Octelium's byte-based pre-flight estimate.
maxOutputTokens rejects a downstream-declared output maximum above the value. It does not insert a limit when the request declares none.
maxTools bounds the number of declared tools.
maxToolSchemaBytes bounds the serialized JSON Schema of one tool.

Zero uses an Octelium default or disables an optional inference limit, depending on the field. Internal hard limits always apply. The input estimate is not provider-accurate and must not be used for billing; read ctx.request.llm.estimateQuality alongside it.

Access Control

LLM-specific request information is stored in ctx.request.llm. The underlying HTTP request and parsed body are available in ctx.request.llm.http and ctx.request.llm.http.bodyMap (read more about HTTP access control here).

authorization:
  inlinePolicies:
    - spec:
        rules:
          - effect: ALLOW
            condition:
              all:
                of:
                  - match: ctx.request.llm.operation == "GENERATE"
                  - match: ctx.request.llm.route == "RESPONSES"
                  - match: ctx.request.llm.model == "gpt-5-mini"
                  - match: ctx.request.llm.stream == false
                  - match: ctx.request.llm.hasTools == false
                  - match: ctx.request.llm.hasImageInput == false
                  - match: ctx.request.llm.hasAudioInput == false
                  - match: ctx.request.llm.maxOutputTokens > 0
                  - match: ctx.request.llm.maxOutputTokens <= 4096
                  - match: '"ai-users" in ctx.user.spec.groups'

The normalized fields are:

protocol, operation, route, model and stream.
estimatedInputTokens, estimateQuality and maxOutputTokens.
hasTools, toolCount and the bounded toolNames list.
inputItemCount, hasImageInput and hasAudioInput.

The values describe the request sent by the downstream before model, prompt or reasoning plugins mutate it. Prefer them over inspecting an untrusted, deeply nested body whenever possible.

Dynamic Configuration

Dynamic configuration can select compatible upstreams, credentials and configuration from identity and normalized LLM context (read more here):

spec:
  mode: LLM
  config:
    upstream:
      url: https://default-provider.example.com/v1
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: default-provider-key
  dynamicConfig:
    configs:
      - name: premium
        upstream:
          url: https://premium-provider.example.com/v1
        llm:
          auth:
            bearer:
              fromSecret: premium-provider-key
          model:
            value: premium-model
    rules:
      - condition:
          all:
            of:
              - match: ctx.request.llm.model == "premium-model"
              - match: '"ai-admins" in ctx.user.spec.groups'
        configName: premium

protocol and limits are global and only take effect in the default configuration. listenHTTP2 and cors are also global. Every selected upstream must speak the global protocol because Octelium does not translate protocols.

HTTP Configuration

Since LLM is HTTP-based, config.llm also supports header and path manipulation, HTTP/2 and CORS (read more here):

llm:
  header:
    addRequestHeaders:
      - key: X-AI-Gateway
        value: octelium
      - key: X-User-ID
        eval: ctx.user.metadata.uid
    removeRequestHeaders:
      - X-Untrusted-Header
    addResponseHeaders:
      - key: X-Content-Source
        value: inference
    removeResponseHeaders:
      - Server
    forwardedMode: DROP
    authorizationMode: DELETE
    host:
      value: inference.internal.example.com
  path:
    addPrefix: /inference
  isUpstreamHTTP2: true
  listenHTTP2: true
  cors:
    allowOriginStringMatch:
      - https://ai-console.example.com
    allowMethods: POST, GET
    allowHeaders: Authorization, Content-Type
    allowCredentials: true
    allowClusterServices: true

Header and path changes apply after the downstream operation is validated. The host can instead be preserved or computed with eval, and forwardedMode can be DROP, OBFUSCATE or TRANSPARENT. isUpstreamHTTP2 affects connections to the provider. listenHTTP2 and cors are global because they affect downstream connections and unauthenticated preflight requests. Enabling cors.allowClusterServices trusts browser applications served by every Service in the Cluster.

Continue with the LLM plugins and LLM visibility references.