Self-Hosted Remote Ollama

Octelium enables you to seamlessly provide secure access to Ollama via both the private client-based mode over WireGuard/QUIC as well as the public clientless secure access (read more about the clientless BeyondCorp access mode here) for lightweight open models such as Google's Gemma 3, DeepSeek R1, Meta's Llama, etc....

Here is a simple example where you can seamlessly deploy an Ollama server as a managed container and serve it as an Octelium Service (read more about managed containers here):

kind: Service
metadata:
  name: ollama
spec:
  port: 11434
  mode: HTTP
  isPublic: true
  config:
    upstream:
      container:
        port: 11434
        image: ollama/ollama
        resourceLimit:
          cpu:
            millicores: 3000
          memory:
            megabytes: 4000

The above configuration is simply a CPU-only mode. If your underlying Kubernetes installation supports requesting and scheduling GPUs (read more here), you can modify the above the configuration to be as follows:

kind: Service
metadata:
  name: ollama
spec:
  port: 11434
  mode: HTTP
  isPublic: true
  config:
    upstream:
      container:
        port: 11434
        image: ollama/ollama
        resourceLimit:
          cpu:
            millicores: 3000
          memory:
            megabytes: 4000
          ext:
            # Change this according to your Kubernetes cluster available values
            nvidia.com/gpu: "1"

note

You might also want to explicitly set OLLAMA_MODELS environment variable (read more here) to a path mounted via a Kubernetes volume in order to enforce persistent storage. Read more about volumes here.

You can now apply the creation of the Service as follows (read more here):

octeliumctl apply /PATH/TO/SERVICE.YAML

You can also serve an Ollama server that's served by a connected User (read more here) as follows:

kind: Service
metadata:
  name: ollama
spec:
  port: 11434
  mode: HTTP
  isPublic: true
  config:
    upstream:
      url: http://localhost:11434
      user: ollama-server

To inform the Ollama client (you can download the client here) with the address of our Service running the Ollama server, you need to set the OLLAMA_HOST environment variable to the address of the Service. For client-based access (read more about connecting to the Cluster here), you need to set the environment variable as follows:

export OLLAMA_HOST=ollama:11434

Now, from your machine, you can run a model like Gemma3 as follows:

ollama run gemma3:1b

You can also access Ollama via the clientless BeyondCorp mode from within your applications written in programming language without having to use the octelium client or any special SDK. For example, you can use the OAuth2 client credential mode (read more here) and use its bearer access token as in the following Python code example:

from ollama import Client

client = Client(
  host='https://ollama.<DOMAIN>',
  headers={'authorization': 'Bearer XXXX'}
)

response = client.chat(model='llama3.2', messages=[
  {
    'role': 'user',
    'content': 'What is Octelium?',
  },
])

note

Ollama is also OpenAI compatible. Read more here.

Regarding authentication for HUMAN Users (read more here), you can seamlessly authenticate to the Cluster via your web browser using an IdentityProvider (read more here). There are currently 3 types:

GitHub OAuth IdentityProvider as shown in detail here
OpenID Connect IdentityProviders (e.g. Okta, Auth0, etc...) as shown here.
SAML 2.0 IdentityProviders (e.g. Okta, Entra ID, etc...) as shown here.

Furthermore, you can can register a FIDO2 Authenticator (e.g. Yubikeys) in order to directly login later via Passkey (read more here) without having to use an IdentityProvider.

note

Alternatively to the OAuth2 client credentials flow Credential, you can also generate an access token Credential and use it directly as a bearer token. Read more here.

Octelium also provides OpenTelemetry-ready, application-layer L7 aware visibility and access logging in real time (see an example for HTTP here). You can read more about visibility here.

Here are a few more features that you might be interested in:

Request/response header manipulation (read more here).
Application layer-aware ABAC access control via policy-as-code using CEL and Open Policy Agent (read more here).
Exposing the API publicly for anonymous access (read more here).
OpenTelemetry-ready, application-layer L7 aware auditing and visibility (read more here).