# Self-Hosted Remote Ollama > Octelium documentation. Canonical page: . Octelium enables you to seamlessly provide secure access to [Ollama](https://github.com/ollama/ollama/) via both the private client-based mode over WireGuard/QUIC as well as the public clientless secure access (read more about the clientless BeyondCorp access mode [here](https://octelium.com/docs/octelium/latest/management/core/service/clientless.md)) for lightweight open models such as Google's [Gemma 3](https://blog.google/technology/developers/gemma-3/), DeepSeek R1, Meta's Llama, Qwen, etc.... Ollama is served via the dedicated `LLM` *Service* mode (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md)), which understands the inference API rather than treating it as generic HTTP traffic. This enables you to apply to your self-hosted models the very same identity-aware access control by model and token limits, token quotas, guardrails, managed prompts and inference-specific visibility that you apply to SaaS providers (read more about using Octelium as an AI gateway [here](https://octelium.com/docs/octelium/latest/management/guide/service/ai/ai-gateway.md)). ## Deploy Ollama Here is a simple example where you can seamlessly deploy an Ollama server as a managed container and serve it as an Octelium *Service* (read more about managed containers [here](https://octelium.com/docs/octelium/latest/management/core/service/managed-containers.md)): ```yaml kind: Service metadata: name: ollama spec: port: 11434 mode: LLM isPublic: true config: upstream: container: port: 11434 image: ollama/ollama command: - /bin/sh - -c - | ollama serve & until ollama list > /dev/null 2>&1; do sleep 1; done ollama pull gemma3:1b wait resourceLimit: cpu: millicores: 3000 memory: megabytes: 4000 llm: protocol: OPENAI ``` Ollama serves the OpenAI-compatible API over the canonical `/v1` routes (read more [here](https://docs.ollama.com/api/openai-compatibility)), so we use the `OPENAI` protocol. The LLM mode only exposes the inference routes of the protocol (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#protocols)) and rejects everything else before it reaches the upstream, including Ollama's native model management API (e.g. `/api/pull` and `/api/delete`). That's why the container pulls the model itself upon startup. The above configuration is simply a CPU-only mode. If your underlying Kubernetes installation supports requesting and scheduling GPUs (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/managed-containers.md#extended-resources)), you can modify the above configuration to be as follows: ```yaml kind: Service metadata: name: ollama spec: port: 11434 mode: LLM isPublic: true config: upstream: container: port: 11434 image: ollama/ollama command: - /bin/sh - -c - | ollama serve & until ollama list > /dev/null 2>&1; do sleep 1; done ollama pull gemma3:1b wait resourceLimit: cpu: millicores: 3000 memory: megabytes: 4000 #!mark(1:3) ext: # Change this according to your Kubernetes cluster available values nvidia.com/gpu: "1" llm: protocol: OPENAI ``` > **Note:** > > The models are stored by default at `/root/.ollama` inside the container. You might want to mount a persistent volume at that path (read more about volumes [here](https://octelium.com/docs/octelium/latest/management/core/service/managed-containers.md#volumes)) or explicitly set the `OLLAMA_MODELS` environment variable (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/managed-containers.md#environment-variables)) to a path mounted via a persistent volume in order to avoid having to pull the models again whenever the container restarts. You can now apply the creation of the *Service* as follows (read more [here](https://octelium.com/docs/octelium/latest/management/core/overview.md)): ```bash octeliumctl apply /PATH/TO/SERVICE.YAML ``` You can also serve an Ollama server that's served by a connected *User* (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/overview.md#remotely-via-a-connected-user)), for example from a GPU-equipped machine behind NAT, as follows: ```yaml kind: Service metadata: name: ollama spec: mode: LLM isPublic: true config: upstream: url: http://localhost:11434 user: ollama-server llm: protocol: OPENAI ``` ## Client-Based Access Once connected to the *Cluster* (read more about connecting to the *Cluster* [here](https://octelium.com/docs/octelium/latest/user/cli/connect.md)), you can access the Ollama OpenAI-compatible API at the base URL `http://ollama:11434/v1` via any OpenAI-compatible client, SDK, coding agent or chat UI. For example, you can use `curl` as follows: ```bash curl http://ollama:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "gemma3:1b", "messages": [{"role": "user", "content": "What is Octelium?"}]}' ``` > **Note:** > > The `ollama` CLI (e.g. `ollama run` and `ollama pull`) uses Ollama's native API rather than the OpenAI-compatible API. Therefore, it cannot be used with an `LLM` *Service*. If you really need the native API, you can still expose Ollama via an ordinary `HTTP` *Service* (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/http.md)), at the cost of losing all the inference-aware features of the `LLM` mode. ## Clientless Access You can also access Ollama via the clientless BeyondCorp mode from within your applications written in any programming language without having to use the `octelium` client or any special SDK. For example, you can use the OAuth2 client credentials flow (read more [here](https://octelium.com/docs/octelium/latest/management/core/credential.md#oauth2-client-credentials)) to obtain an Octelium access token and use it as the API key of the ordinary OpenAI SDK as in the following Python code example: ```python import os import requests from openai import OpenAI domain = os.environ["OCTELIUM_DOMAIN"] # Obtain an Octelium access token via the OAuth2 client credentials flow token_response = requests.post( f"https://{domain}/oauth2/token", data={ "grant_type": "client_credentials", "client_id": os.environ["OCTELIUM_CLIENT_ID"], "client_secret": os.environ["OCTELIUM_CLIENT_SECRET"], }, ) token_response.raise_for_status() client = OpenAI( base_url=f"https://ollama.{domain}/v1", api_key=token_response.json()["access_token"], ) response = client.chat.completions.create( model="gemma3:1b", messages=[{"role": "user", "content": "What is Octelium?"}], max_tokens=1024, ) print(response.choices[0].message.content) ``` The OpenAI SDK sends the Octelium access token to the *Service* while Ollama itself never needs to be exposed or to be aware of the *Cluster* at all. Read more about using OAuth2 client credentials in other programming languages [here](https://octelium.com/docs/octelium/latest/user/clientless/oauth2.md). > **Note:** > > Alternatively to the OAuth2 client credentials flow *Credential*, you can also > generate an access token *Credential* and use it directly as a bearer token. > Read more [here](https://octelium.com/docs/octelium/latest/management/core/credential.md#access-tokens). Regarding authentication for `HUMAN` *Users* (read more [here](https://octelium.com/docs/octelium/latest/management/core/user.md#types)), you can seamlessly authenticate to the *Cluster* via your web browser using an *IdentityProvider* (read more [here](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md)). There are currently 3 types: - GitHub OAuth *IdentityProvider* as shown in detail [here](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md#github-oauth2) - OpenID Connect *IdentityProviders* (e.g. Okta, Auth0, etc...) as shown [here](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md#openid-connect). - SAML 2.0 *IdentityProviders* (e.g. Okta, Entra ID, etc...) as shown [here](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md#saml-20). Furthermore, you can register a FIDO2 *Authenticator* (e.g. Yubikeys) in order to directly login later via Passkey (read more [here](https://octelium.com/docs/octelium/latest/management/core/authenticator.md#passkey-login)) without having to use an *IdentityProvider*. ## Access Control The LLM mode exposes the normalized inference request information under `ctx.request.llm` to your *Policies* (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#access-control)). For example, the following *Policy* only allows members of the `ai-users` group to generate output using the `gemma3:1b` model with an explicit output maximum: ```yaml authorization: inlinePolicies: - spec: rules: - effect: ALLOW condition: all: of: - match: '"ai-users" in ctx.user.spec.groups' - match: ctx.request.llm.operation == "GENERATE" - match: ctx.request.llm.model == "gemma3:1b" - match: ctx.request.llm.maxOutputTokens > 0 - match: ctx.request.llm.maxOutputTokens <= 4096 ``` ## Token Quotas The GPU time of your self-hosted models is typically the scarcest resource. Instead of only limiting the request count, you can limit the tokens consumed by every *User* (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#token-rate-limit)) as follows: ```yaml llm: protocol: OPENAI limits: maxOutputTokens: 4096 plugins: - name: hourly-user-budget condition: match: ctx.request.llm.operation == "GENERATE" tokenRateLimit: scope: TOTAL key: perUser: true limit: 50000 window: hours: 1 defaultOutputTokens: 1024 denyMessage: The hourly token budget has been exhausted ``` ## Semantic Caching Ollama also serves embedding models over the OpenAI-compatible `/v1/embeddings` route. Therefore, you can enable semantic caching (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-cache)) using the same Ollama server as the embedding backend and keep your entire inference stack self-hosted. You only need to additionally pull an embedding model such as `nomic-embed-text` in the container: ```yaml llm: protocol: OPENAI embedding: source: currentUpstream: true model: nomic-embed-text plugins: - name: user-cache condition: match: ctx.request.llm.operation == "GENERATE" semanticCache: scope: perUser: true minSimilarity: 0.92 ttl: hours: 6 maxSize: 1048576 ``` A cache hit avoids an inference request entirely and therefore consumes no GPU time and no token quota. ## Visibility Octelium also provides OpenTelemetry-ready, inference-aware visibility and access logging in real time that includes the authenticated identity, the operation, the requested and served model, the provider-reported token usage, the finish reason and the time to first token (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/visibility.md)). You can read more about visibility [here](https://octelium.com/docs/octelium/latest/management/core/visibility.md). Here are a few more features that you might be interested in: - Managed system instructions, message injection, tool filtering and reasoning controls (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md)). - Request and response guardrails for secrets, personal information and custom patterns (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#guardrail)). - Semantic routing between different models according to the meaning of the request (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-router)). - Dynamic routing to different upstreams and models according to identity and request context (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/dynamic-config.md)). - Exposing the API publicly for anonymous access (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/anonymous-access.md)).