Self-Hosted Remote Ollama

Octelium enables you to seamlessly provide secure access to Ollama via both the private client-based mode over WireGuard/QUIC as well as the public clientless secure access (read more about the clientless BeyondCorp access mode here) for lightweight open models such as Google's Gemma 3, DeepSeek R1, Meta's Llama, Qwen, etc....

Ollama is served via the dedicated LLM Service mode (read more here), which understands the inference API rather than treating it as generic HTTP traffic. This enables you to apply to your self-hosted models the very same identity-aware access control by model and token limits, token quotas, guardrails, managed prompts and inference-specific visibility that you apply to SaaS providers (read more about using Octelium as an AI gateway here).

Deploy Ollama

Here is a simple example where you can seamlessly deploy an Ollama server as a managed container and serve it as an Octelium Service (read more about managed containers here):

kind: Service
metadata:
  name: ollama
spec:
  port: 11434
  mode: LLM
  isPublic: true
  config:
    upstream:
      container:
        port: 11434
        image: ollama/ollama
        command:
          - /bin/sh
          - -c
          - |
            ollama serve &
            until ollama list > /dev/null 2>&1; do sleep 1; done
            ollama pull gemma3:1b
            wait
        resourceLimit:
          cpu:
            millicores: 3000
          memory:
            megabytes: 4000
    llm:
      protocol: OPENAI

Ollama serves the OpenAI-compatible API over the canonical /v1 routes (read more here), so we use the OPENAI protocol. The LLM mode only exposes the inference routes of the protocol (read more here) and rejects everything else before it reaches the upstream, including Ollama's native model management API (e.g. /api/pull and /api/delete). That's why the container pulls the model itself upon startup.

The above configuration is simply a CPU-only mode. If your underlying Kubernetes installation supports requesting and scheduling GPUs (read more here), you can modify the above configuration to be as follows:

kind: Service
metadata:
  name: ollama
spec:
  port: 11434
  mode: LLM
  isPublic: true
  config:
    upstream:
      container:
        port: 11434
        image: ollama/ollama
        command:
          - /bin/sh
          - -c
          - |
            ollama serve &
            until ollama list > /dev/null 2>&1; do sleep 1; done
            ollama pull gemma3:1b
            wait
        resourceLimit:
          cpu:
            millicores: 3000
          memory:
            megabytes: 4000
          ext:
            # Change this according to your Kubernetes cluster available values
            nvidia.com/gpu: "1"
    llm:
      protocol: OPENAI

note

The models are stored by default at /root/.ollama inside the container. You might want to mount a persistent volume at that path (read more about volumes here) or explicitly set the OLLAMA_MODELS environment variable (read more here) to a path mounted via a persistent volume in order to avoid having to pull the models again whenever the container restarts.

You can now apply the creation of the Service as follows (read more here):

octeliumctl apply /PATH/TO/SERVICE.YAML

You can also serve an Ollama server that's served by a connected User (read more here), for example from a GPU-equipped machine behind NAT, as follows:

kind: Service
metadata:
  name: ollama
spec:
  mode: LLM
  isPublic: true
  config:
    upstream:
      url: http://localhost:11434
      user: ollama-server
    llm:
      protocol: OPENAI

Client-Based Access

Once connected to the Cluster (read more about connecting to the Cluster here), you can access the Ollama OpenAI-compatible API at the base URL http://ollama:11434/v1 via any OpenAI-compatible client, SDK, coding agent or chat UI. For example, you can use curl as follows:

curl http://ollama:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemma3:1b", "messages": [{"role": "user", "content": "What is Octelium?"}]}'

note

The ollama CLI (e.g. ollama run and ollama pull) uses Ollama's native API rather than the OpenAI-compatible API. Therefore, it cannot be used with an LLM Service. If you really need the native API, you can still expose Ollama via an ordinary HTTP Service (read more here), at the cost of losing all the inference-aware features of the LLM mode.

Clientless Access

You can also access Ollama via the clientless BeyondCorp mode from within your applications written in any programming language without having to use the octelium client or any special SDK. For example, you can use the OAuth2 client credentials flow (read more here) to obtain an Octelium access token and use it as the API key of the ordinary OpenAI SDK as in the following Python code example:

import os

import requests
from openai import OpenAI

domain = os.environ["OCTELIUM_DOMAIN"]

# Obtain an Octelium access token via the OAuth2 client credentials flow
token_response = requests.post(
    f"https://{domain}/oauth2/token",
    data={
        "grant_type": "client_credentials",
        "client_id": os.environ["OCTELIUM_CLIENT_ID"],
        "client_secret": os.environ["OCTELIUM_CLIENT_SECRET"],
    },
)
token_response.raise_for_status()

client = OpenAI(
    base_url=f"https://ollama.{domain}/v1",
    api_key=token_response.json()["access_token"],
)

response = client.chat.completions.create(
    model="gemma3:1b",
    messages=[{"role": "user", "content": "What is Octelium?"}],
    max_tokens=1024,
)

print(response.choices[0].message.content)

The OpenAI SDK sends the Octelium access token to the Service while Ollama itself never needs to be exposed or to be aware of the Cluster at all. Read more about using OAuth2 client credentials in other programming languages here.

note

Alternatively to the OAuth2 client credentials flow Credential, you can also generate an access token Credential and use it directly as a bearer token. Read more here.

Regarding authentication for HUMAN Users (read more here), you can seamlessly authenticate to the Cluster via your web browser using an IdentityProvider (read more here). There are currently 3 types:

GitHub OAuth IdentityProvider as shown in detail here
OpenID Connect IdentityProviders (e.g. Okta, Auth0, etc...) as shown here.
SAML 2.0 IdentityProviders (e.g. Okta, Entra ID, etc...) as shown here.

Furthermore, you can register a FIDO2 Authenticator (e.g. Yubikeys) in order to directly login later via Passkey (read more here) without having to use an IdentityProvider.

Access Control

The LLM mode exposes the normalized inference request information under ctx.request.llm to your Policies (read more here). For example, the following Policy only allows members of the ai-users group to generate output using the gemma3:1b model with an explicit output maximum:

authorization:
  inlinePolicies:
    - spec:
        rules:
          - effect: ALLOW
            condition:
              all:
                of:
                  - match: '"ai-users" in ctx.user.spec.groups'
                  - match: ctx.request.llm.operation == "GENERATE"
                  - match: ctx.request.llm.model == "gemma3:1b"
                  - match: ctx.request.llm.maxOutputTokens > 0
                  - match: ctx.request.llm.maxOutputTokens <= 4096

Token Quotas

The GPU time of your self-hosted models is typically the scarcest resource. Instead of only limiting the request count, you can limit the tokens consumed by every User (read more here) as follows:

llm:
  protocol: OPENAI
  limits:
    maxOutputTokens: 4096
  plugins:
    - name: hourly-user-budget
      condition:
        match: ctx.request.llm.operation == "GENERATE"
      tokenRateLimit:
        scope: TOTAL
        key:
          perUser: true
        limit: 50000
        window:
          hours: 1
        defaultOutputTokens: 1024
        denyMessage: The hourly token budget has been exhausted

Semantic Caching

Ollama also serves embedding models over the OpenAI-compatible /v1/embeddings route. Therefore, you can enable semantic caching (read more here) using the same Ollama server as the embedding backend and keep your entire inference stack self-hosted. You only need to additionally pull an embedding model such as nomic-embed-text in the container:

llm:
  protocol: OPENAI
  embedding:
    source:
      currentUpstream: true
    model: nomic-embed-text
  plugins:
    - name: user-cache
      condition:
        match: ctx.request.llm.operation == "GENERATE"
      semanticCache:
        scope:
          perUser: true
        minSimilarity: 0.92
        ttl:
          hours: 6
        maxSize: 1048576

A cache hit avoids an inference request entirely and therefore consumes no GPU time and no token quota.

Visibility

Octelium also provides OpenTelemetry-ready, inference-aware visibility and access logging in real time that includes the authenticated identity, the operation, the requested and served model, the provider-reported token usage, the finish reason and the time to first token (read more here). You can read more about visibility here.

Here are a few more features that you might be interested in:

Managed system instructions, message injection, tool filtering and reasoning controls (read more here).
Request and response guardrails for secrets, personal information and custom patterns (read more here).
Semantic routing between different models according to the meaning of the request (read more here).
Dynamic routing to different upstreams and models according to identity and request context (read more here).
Exposing the API publicly for anonymous access (read more here).