# Overview > Octelium documentation. Canonical page: . The `LLM` mode turns an Octelium *Service* into an identity-aware LLM gateway. It validates inference API requests, makes normalized LLM information available to *Policies* and dynamic configuration, injects upstream credentials and records inference-specific access logs. It also provides inference-aware [plugins](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md) for prompts, tools, guardrails, model selection, reasoning, token rate limiting, semantic caching and semantic routing. Here is a simple OpenAI gateway: ```bash octeliumctl create secret openai-api-key ``` ```yaml kind: Service metadata: name: openai spec: mode: LLM isPublic: true config: upstream: url: https://api.openai.com # !mark(1:5) llm: protocol: OPENAI auth: bearer: fromSecret: openai-api-key ``` The `isPublic` field enables public clientless access to the *Service* (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/clientless.md)). An authorized workload can use the *Service* URL as the base URL of an ordinary OpenAI client and use its Octelium access token instead of the provider API key. ## Protocols The `protocol` field describes the inference API spoken by downstream clients and, unless a [translation](#protocol-translation) is set, by the upstream as well. It defaults to `OPENAI`. Octelium validates and normalizes the following protocols: - `OPENAI` supports `POST /v1/chat/completions`, `POST /v1/responses`, `POST /v1/completions`, `POST /v1/embeddings`, `POST /v1/moderations`, `GET /v1/models` and `GET /v1/models/{model}`. - `ANTHROPIC` supports `POST /v1/messages`, `POST /v1/messages/count_tokens`, `GET /v1/models` and `GET /v1/models/{model}`. - `GEMINI` supports content generation, streaming content generation, token counting and single or batch embeddings over the Gemini Developer API `/v1beta` routes. - `BEDROCK` supports the Amazon Bedrock Runtime Converse and InvokeModel routes. Converse has normalized inference semantics. InvokeModel carries a model-native body, so it is proxied as raw inference without body parsing or inference-specific plugin processing. Body-carrying operations require an `application/json` media type and a JSON object. Unsupported paths, methods and streaming combinations are rejected before reaching the upstream. > **Note:** > > By default, the downstream and upstream of a *Service* speak the same protocol. The generation operation can be translated to an upstream that speaks another protocol as shown [below](#protocol-translation). OpenAI-compatible servers such as vLLM and Ollama are supported when they expose the canonical routes. Azure OpenAI deployment routes and Vertex AI project-scoped routes are not the `OPENAI` and `GEMINI` protocols described here. ## Operations and Routes `ctx.request.llm.operation` describes what a request does independently of the provider protocol: - `GENERATE` generates output from an input. - `EMBED` generates vector embeddings. - `MODERATE` classifies an input for safety. - `COUNT_TOKENS` counts input tokens without generating output. - `LIST_MODELS` and `GET_MODEL` discover models. - `RAW_INFERENCE` represents a model-native Bedrock InvokeModel request whose body Octelium does not parse. `ctx.request.llm.route` retains the exact API surface. Its values are `CHAT_COMPLETIONS`, `RESPONSES`, `COMPLETIONS`, `EMBEDDINGS`, `MODERATIONS`, `MODELS_LIST`, `MODELS_GET`, `MESSAGES`, `COUNT_TOKENS`, `GENERATE_CONTENT`, `EMBED_CONTENT`, `CONVERSE` and `INVOKE_MODEL`. Use the operation in rules that should work across protocols and the route when the exact API shape matters. ## Upstream Base Paths The path in `upstream.url` acts as the provider base path. For `OPENAI`, `ANTHROPIC` and `GEMINI`, Octelium removes the leading version segment from the canonical downstream route and appends the remainder to that base path. The following example forwards `/v1/chat/completions` to `/api/v1/chat/completions`: ```yaml spec: mode: LLM config: upstream: url: https://openrouter.ai/api/v1 llm: protocol: OPENAI auth: bearer: fromSecret: openrouter-api-key ``` When `upstream.url` has no path, the canonical downstream path is preserved. `BEDROCK` has no version prefix and appends its route as is. The `path` configuration or a `path` plugin can handle providers with another layout. ## Protocol Translation The `translation` field lets downstreams keep using whatever protocol and SDK they already use regardless of the protocol spoken by the upstream. The following example serves the `OPENAI` protocol to downstreams while the upstream is Anthropic: ```bash octeliumctl create secret anthropic-api-key ``` ```yaml kind: Service metadata: name: claude spec: mode: LLM config: upstream: url: https://api.anthropic.com llm: protocol: OPENAI # !mark(1:3) translation: upstreamProtocol: ANTHROPIC defaultMaxOutputTokens: 4096 auth: custom: header: x-api-key value: fromSecret: anthropic-api-key ``` `protocol` is the protocol served to downstreams and `upstreamProtocol` is the protocol spoken by the upstream. No translation is performed when `upstreamProtocol` is unset or equal to `protocol`. The `OPENAI`, `ANTHROPIC`, `GEMINI` and `BEDROCK` protocols can be translated to one another for the generation operation, in both streaming and non-streaming modes: - `OPENAI` translates the `CHAT_COMPLETIONS` route. - `ANTHROPIC` translates the `MESSAGES` route. - `GEMINI` translates the `GENERATE_CONTENT` route. - `BEDROCK` translates the `CONVERSE` route. Other operations such as embeddings, moderations, token counting, model discovery and Bedrock InvokeModel have no counterpart in the other protocols and are rejected rather than proxied. The upstream base path rules described [above](#upstream-base-paths) apply to the upstream protocol route. The `anthropic-version` header is automatically set for an `ANTHROPIC` upstream when it is not explicitly set. Octelium translates a bounded subset that the protocols have in common: text, images, system instructions, multi-turn history, client tools, tool calls and results, tool choice, stop sequences, output limits, sampling, streaming and usage. Provider-specific features that no other protocol defines (e.g. OpenAI structured outputs and logprobs, Anthropic thinking blocks, assistant prefills, citations and server tools, Gemini safety settings and built-in tools, and Bedrock guardrail configuration) are rejected instead of being silently dropped. Prompt cache controls that the target protocol does not define are dropped since they only affect cost. Tool schemas are proxied verbatim. `defaultMaxOutputTokens` is used when the downstream protocol allows omitting the output limit while the upstream protocol requires one, which is the case of an `ANTHROPIC` upstream. It only applies to requests that declare no output limit. Zero uses a default of `4096`. It is bounded by `limits.maxOutputTokens`, and a reasoning token budget decided by the *Service* is added on top of it. `ctx.request.llm.maxOutputTokens` still reports zero for such requests. `GEMINI` and `BEDROCK` name the model in the request path. A translated request is therefore addressed to the model resolved by the *Service* (i.e. by `model`, a model plugin or a semantic router), or otherwise to the model requested by the downstream. A request that names no model or names a model that the upstream protocol cannot address is rejected. The translation is applied after every inference-specific plugin on the request and before every one of them on the response. Policies, plugins, semantic caches and access logs therefore always see the downstream protocol. A downstream reasoning configuration is only translated when both protocols express reasoning in the same terms. Use the `reasoning` field (read more [below](#reasoning)) or a reasoning plugin to control reasoning for translated requests. > **Note:** > > `translation` is global and only takes effect in the default configuration. Gemini tool-call signatures that must be returned on the next turn are retained by Octelium, bound to the *Session* and the upstream model, and restored into later translated requests. ## Secretless Access The `auth` field supports bearer tokens, custom API key headers, basic authentication, OAuth2 client credentials and AWS Signature Version 4 (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/secretless.md)). The credential used to access the Octelium *Service* is independent from the provider credential. Downstream credentials are never forwarded to the provider. ### Custom API Key Anthropic uses an API key in `x-api-key` and requires a version header: ```yaml llm: protocol: ANTHROPIC auth: custom: header: x-api-key value: fromSecret: anthropic-api-key header: addRequestHeaders: - key: anthropic-version value: "2023-06-01" ``` Gemini can use the same authentication type with `header: x-goog-api-key`. ### OAuth2 Client Credentials ```yaml llm: auth: oauth2ClientCredentials: clientID: llm-gateway clientSecret: fromSecret: provider-client-secret tokenURL: https://identity.example.com/oauth2/token scopes: - inference ``` ### Basic Authentication ```yaml llm: auth: basic: username: inference-client password: fromSecret: provider-password ``` ### AWS Signature Version 4 ```yaml spec: mode: LLM config: upstream: url: https://bedrock-runtime.us-east-1.amazonaws.com llm: protocol: BEDROCK auth: sigv4: accessKeyID: AKIAEXAMPLE secretAccessKey: fromSecret: aws-secret-access-key region: us-east-1 service: bedrock ``` ## Model Selection By default, the model requested by the downstream is preserved. `model` can replace it with a static value, a CEL result or a Rego result. ### Static Model ```yaml llm: model: # !mark value: gpt-5-mini ``` ### CEL ```yaml llm: model: # !mark eval: '"ai-admins" in ctx.user.spec.groups ? "gpt-5" : "gpt-5-mini"' ``` The CEL expression must return a string. An empty string preserves the downstream model. ### Open Policy Agent ```yaml llm: model: # !mark opa: | package octelium.eval default result := "gpt-5-mini" result := "gpt-5" if { "ai-admins" in input.ctx.user.spec.groups } ``` The Rego script must produce a string through `result`. An empty result preserves the requested model. The normalized `ctx.request.llm.model` field always contains the downstream-requested value, even when the upstream model is replaced. Conditional model selection is also available as a [plugin](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#model-and-reasoning). ## Reasoning The `reasoning` field makes the *Service*, rather than the downstream, decide how much a reasoning model is allowed to think. It overwrites the downstream configuration for operations that support reasoning and is a no-op for other operations. ### Portable Level ```yaml llm: reasoning: # !mark level: MEDIUM ``` The portable levels are `NONE`, `MINIMAL`, `LOW`, `MEDIUM`, `HIGH`, `XHIGH` and `MAX`. Octelium selects the strongest model-supported configuration that does not exceed the configured level. It never resolves upwards or silently drops an incompatible restriction. ### Token Budget ```yaml llm: reasoning: # !mark tokenBudget: 4096 ``` `tokenBudget` is exact and only works with models that accept numeric reasoning budgets. A model that only accepts ordinal effort is rejected. Zero is invalid; use `level: NONE` to disable reasoning. ### Provider-Native Effort ```yaml llm: reasoning: # !mark effort: xhigh ``` `effort` is passed as an ordinal provider-native value. It is the escape hatch for an effort not represented by a portable level and is not translated to a numeric budget. ### CEL ```yaml llm: reasoning: # !mark eval: 'ctx.request.llm.model == "gpt-5" ? "HIGH" : "NONE"' ``` The expression returns a portable level such as `HIGH`, a numeric budget such as `4096`, or a provider-native effort. An empty result preserves the downstream reasoning configuration. ### Open Policy Agent ```yaml llm: reasoning: # !mark opa: | package octelium.eval default result := "LOW" result := "MAX" if { "ai-admins" in input.ctx.user.spec.groups } ``` Conditional reasoning configuration is also available as a [plugin](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#model-and-reasoning). The last matching reasoning plugin overwrites this default. ## Embeddings Semantic caching and routing require an embedding configuration. Define it once in `llm.embedding`, or override it within an individual semantic plugin. The current upstream can generate embeddings for `OPENAI` and `GEMINI` *Services*: ```yaml llm: embedding: source: currentUpstream: true model: text-embedding-3-small dimensions: 1024 ``` Use a separate `OPENAI` or `GEMINI` embedding API when the main upstream does not provide compatible embeddings: ```yaml llm: embedding: source: upstream: url: https://embeddings.example.com/v1 protocol: OPENAI auth: bearer: fromSecret: embeddings-api-key model: text-embedding-3-small ``` Zero `dimensions` uses the model default. Changing the model or dimensions makes vectors stored with the previous configuration unreachable. The embedding backend receives prompt-derived content after request guardrails have run. ## Limits ```yaml llm: limits: maxRequestBytes: 8388608 maxStreamEventBytes: 262144 maxEstimatedInputTokens: 100000 maxOutputTokens: 8192 maxTools: 64 maxToolSchemaBytes: 65536 ``` - `maxRequestBytes` bounds the JSON request body buffered and parsed by Octelium. - `maxStreamEventBytes` bounds inspection of an individual streamed event. A larger event is forwarded but not inspected. - `maxEstimatedInputTokens` rejects a request above Octelium's byte-based pre-flight estimate. - `maxOutputTokens` rejects a downstream-declared output maximum above the value. It does not insert a limit when the request declares none. - `maxTools` bounds the number of declared tools. - `maxToolSchemaBytes` bounds the serialized JSON Schema of one tool. Zero uses an Octelium default or disables an optional inference limit, depending on the field. Internal hard limits always apply. The input estimate is not provider-accurate and must not be used for billing; read `ctx.request.llm.estimateQuality` alongside it. ## Access Control LLM-specific request information is stored in `ctx.request.llm`. The underlying HTTP request and parsed body are available in `ctx.request.llm.http` and `ctx.request.llm.http.bodyMap` (read more about HTTP access control [here](https://octelium.com/docs/octelium/latest/management/core/service/http.md#access-control)). ```yaml authorization: inlinePolicies: - spec: rules: - effect: ALLOW condition: all: of: - match: ctx.request.llm.operation == "GENERATE" - match: ctx.request.llm.route == "RESPONSES" - match: ctx.request.llm.model == "gpt-5-mini" - match: ctx.request.llm.stream == false - match: ctx.request.llm.hasTools == false - match: ctx.request.llm.hasImageInput == false - match: ctx.request.llm.hasAudioInput == false - match: ctx.request.llm.maxOutputTokens > 0 - match: ctx.request.llm.maxOutputTokens <= 4096 - match: '"ai-users" in ctx.user.spec.groups' ``` The normalized fields are: - `protocol`, `operation`, `route`, `model` and `stream`. - `estimatedInputTokens`, `estimateQuality` and `maxOutputTokens`. - `hasTools`, `toolCount` and the bounded `toolNames` list. - `inputItemCount`, `hasImageInput` and `hasAudioInput`. The values describe the request sent by the downstream before model, prompt or reasoning plugins mutate it. Prefer them over inspecting an untrusted, deeply nested body whenever possible. ## Dynamic Configuration Dynamic configuration can select compatible upstreams, credentials and configuration from identity and normalized LLM context (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/dynamic-config.md)): ```yaml spec: mode: LLM config: upstream: url: https://default-provider.example.com/v1 llm: protocol: OPENAI auth: bearer: fromSecret: default-provider-key dynamicConfig: configs: - name: premium upstream: url: https://premium-provider.example.com/v1 llm: auth: bearer: fromSecret: premium-provider-key model: value: premium-model rules: - condition: all: of: - match: ctx.request.llm.model == "premium-model" - match: '"ai-admins" in ctx.user.spec.groups' configName: premium ``` `protocol`, `translation` and `limits` are global and only take effect in the default configuration. `listenHTTP2` and `cors` are also global. Every selected upstream must speak the global upstream protocol, which is `translation.upstreamProtocol` when set and `protocol` otherwise. ## HTTP Configuration Since LLM is HTTP-based, `config.llm` also supports header and path manipulation, HTTP/2 and CORS (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/http.md)): ```yaml llm: header: addRequestHeaders: - key: X-AI-Gateway value: octelium - key: X-User-ID eval: ctx.user.metadata.uid removeRequestHeaders: - X-Untrusted-Header addResponseHeaders: - key: X-Content-Source value: inference removeResponseHeaders: - Server forwardedMode: DROP authorizationMode: DELETE host: value: inference.internal.example.com path: addPrefix: /inference isUpstreamHTTP2: true listenHTTP2: true cors: allowOriginStringMatch: - https://ai-console.example.com allowMethods: POST, GET allowHeaders: Authorization, Content-Type allowCredentials: true allowClusterServices: true ``` Header and path changes apply after the downstream operation is validated. The host can instead be preserved or computed with `eval`, and `forwardedMode` can be `DROP`, `OBFUSCATE` or `TRANSPARENT`. `isUpstreamHTTP2` affects connections to the provider. `listenHTTP2` and `cors` are global because they affect downstream connections and unauthenticated preflight requests. Enabling `cors.allowClusterServices` trusts browser applications served by every *Service* in the *Cluster*. Continue with the [LLM plugins](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md) and [LLM visibility](https://octelium.com/docs/octelium/latest/management/core/service/llm/visibility.md) references.