Self-Hosted Secure Remote Access to llama.cpp

Octelium enables you to seamlessly provide secure access to llama.cpp server via both the private client-based mode over WireGuard/QUIC as well as the public clientless secure access (read more about the clientless access mode here) for lightweight open models such as Google's Gemma 3, DeepSeek R1, Meta's Llama, Qwen, etc....

The llama.cpp server is served via the dedicated LLM Service mode (read more here), which understands the inference API rather than treating it as generic HTTP traffic. This enables you to apply to your self-hosted models the very same identity-aware access control by model and token limits, token quotas, guardrails, managed prompts and inference-specific visibility that you apply to SaaS providers (read more about using Octelium as an AI gateway here).

Deploy llama.cpp

Here is a simple example where you can seamlessly deploy a llama.cpp server as a managed container and serve it as an Octelium Service (read more about managed containers here):

kind: Service
metadata:
  name: llama
spec:
  port: 8080
  mode: LLM
  isPublic: true
  config:
    upstream:
      container:
        port: 8080
        image: ghcr.io/ggml-org/llama.cpp:server
        args:
          - --model-url
          - https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf
          - --alias
          - qwen2.5-0.5b-instruct
          - --host
          - 0.0.0.0
          - --port
          - "8080"
          - -c
          - "2048"
        resourceLimit:
          cpu:
            millicores: 2000
          memory:
            megabytes: 3000
    llm:
      protocol: OPENAI

The llama.cpp server serves the OpenAI-compatible API over the canonical /v1 routes, so we use the OPENAI protocol (read more about the supported protocols here). The --alias flag sets a stable model name that is reported by the server and that you can use in your Policies regardless of the underlying GGUF file name. Note that the server must listen on 0.0.0.0 in order to be reachable.

If your underlying Kubernetes installation supports requesting and scheduling GPUs (read more here), you can modify the above configuration to be as follows:

kind: Service
metadata:
  name: llama
spec:
  port: 8080
  mode: LLM
  isPublic: true
  config:
    upstream:
      container:
        port: 8080
        image: ghcr.io/ggml-org/llama.cpp:server-cuda
        args:
          - --model-url
          - https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf
          - --alias
          - qwen2.5-0.5b-instruct
          - --host
          - 0.0.0.0
          - --port
          - "8080"
          - -c
          - "2048"
        resourceLimit:
          cpu:
            millicores: 2000
          memory:
            megabytes: 3000
          ext:
            # Change this according to your Kubernetes cluster available values
            nvidia.com/gpu: "1"
    llm:
      protocol: OPENAI

You can now apply the creation of the Service as follows (read more here):

octeliumctl apply /PATH/TO/SERVICE.YAML

You can also serve a llama.cpp server that's served by a connected User (read more here), for example from a GPU-equipped machine behind NAT, by simply setting the upstream to url: http://localhost:8080 and user: <USER> instead of the managed container.

Secretless Access

The llama.cpp server can itself require an API key via its --api-key flag or its LLAMA_API_KEY environment variable. This protects the server even from workloads that can reach it directly from within your network. Octelium can inject that API key on-the-fly (read more about secretless access here) so that neither your Users nor your applications ever see it. First, we create a Secret for the API key (read more here):

octeliumctl create secret llama-api-key

Now we use the same Secret both as the server's API key and as the upstream bearer credential of the Service as follows:

kind: Service
metadata:
  name: llama
spec:
  port: 8080
  mode: LLM
  isPublic: true
  config:
    upstream:
      container:
        port: 8080
        image: ghcr.io/ggml-org/llama.cpp:server
        args:
          - --model-url
          - https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf
          - --alias
          - qwen2.5-0.5b-instruct
          - --host
          - 0.0.0.0
          - --port
          - "8080"
          - -c
          - "2048"
        env:
          - name: LLAMA_API_KEY
            fromSecret: llama-api-key
        resourceLimit:
          cpu:
            millicores: 2000
          memory:
            megabytes: 3000
    llm:
      protocol: OPENAI
      auth:
        bearer:
          fromSecret: llama-api-key

The downstream client authenticates to Octelium. Octelium authenticates separately to the llama.cpp server and never exposes the API key to the client.

Client-Based Access

Now you can access the llama.cpp server as an ordinary OpenAI-compatible server either via the client-based mode via octelium connect or via the clientless mode via OAuth2 client credentials or directly via bearer access tokens. In the client-based access mode, Users can simply access the Service at the base URL http://llama:8080/v1 via any OpenAI-compatible client, SDK, coding agent or chat UI. For example:

curl http://llama:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is Octelium?"}]}'

note

The LLM mode only exposes the inference routes of the protocol and rejects every other route before it reaches the upstream. This includes the non-OpenAI routes of the llama.cpp server such as /completion, /tokenize, /slots and its built-in web UI.

Clientless Access

As for the clientless mode, WORKLOAD Users can access the Service via the standard OAuth2 client credentials in order for your workloads that can be written in any programming language to access the Service without having to use any special SDKs or have access to external clients. All you need is to create an OAUTH2 Credential as illustrated here. Now, here is an example written in Typescript:

import OpenAI from "openai";
import { OAuth2Client } from "@badgateway/oauth2-client";

async function main() {
  const oauth2Client = new OAuth2Client({
    server: "https://<DOMAIN>/",
    clientId: "spxg-cdyx",
    clientSecret: "AQpAzNmdEcPIfWYR2l2zLjMJm....",
    tokenEndpoint: "/oauth2/token",
    authenticationMethod: "client_secret_post",
  });

  const oauth2Creds = await oauth2Client.clientCredentials();
  const client = new OpenAI({
    apiKey: oauth2Creds.accessToken,
    baseURL: "https://llama.<DOMAIN>/v1",
  });

  const chatCompletion = await client.chat.completions.create({
    model: "qwen2.5-0.5b-instruct",
    messages: [
      { role: "user", content: "How do I write a Golang HTTP reverse proxy?" },
    ],
    max_tokens: 1024,
  });

  console.log(chatCompletion.choices[0].message.content);
}

main();

Note that the base URL includes the /v1 prefix of the canonical OpenAI routes. You can read more about using OAuth2 client credentials in other programming languages here.

Regarding authentication for HUMAN Users (read more here), you can seamlessly authenticate to the Cluster via your web browser using an IdentityProvider (read more here). There are currently 3 types:

GitHub OAuth IdentityProvider as shown in detail here
OpenID Connect IdentityProviders (e.g. Okta, Auth0, etc...) as shown here.
SAML 2.0 IdentityProviders (e.g. Okta, Entra ID, etc...) as shown here.

Furthermore, you can register a FIDO2 Authenticator (e.g. Yubikeys) in order to directly login later via Passkey (read more here) without having to use an IdentityProvider.

Access Control and Guardrails

The LLM mode exposes the normalized inference request information under ctx.request.llm to your Policies (read more here). For example, the following Policy only allows members of the ai-users group to generate output from the served model without tools and with an explicit output maximum:

authorization:
  inlinePolicies:
    - spec:
        rules:
          - effect: ALLOW
            condition:
              all:
                of:
                  - match: '"ai-users" in ctx.user.spec.groups'
                  - match: ctx.request.llm.operation == "GENERATE"
                  - match: ctx.request.llm.model == "qwen2.5-0.5b-instruct"
                  - match: ctx.request.llm.hasTools == false
                  - match: ctx.request.llm.maxOutputTokens > 0
                  - match: ctx.request.llm.maxOutputTokens <= 2048

You can also use guardrails (read more here) in order to block detected credentials and redact email addresses before the prompt reaches the model, as well as bound the request size and the requested output (read more about limits here) as follows:

llm:
  protocol: OPENAI
  limits:
    maxEstimatedInputTokens: 2048
    maxOutputTokens: 2048
  plugins:
    - name: protect-input
      condition:
        match: ctx.request.llm.operation == "GENERATE"
      guardrail:
        leg: REQUEST
        scopes:
          - CONTENT
          - INSTRUCTIONS
          - TOOL_RESULTS
        patterns:
          - secrets: {}
            action: DENY
          - type: EMAIL
            action: REDACT
        denyMessage: The request contains restricted content

Visibility

Octelium also provides OpenTelemetry-ready, inference-aware visibility and access logging in real time that includes the authenticated identity, the operation, the requested and served model, the provider-reported token usage, the finish reason and the time to first token (read more here). You can read more about visibility here.

Here are a few more features that you might be interested in:

Token-consumption quotas per User, per Session or per tenant (read more here).
Managed system instructions, message injection, tool filtering and reasoning controls (read more here).
Semantic caching and semantic routing between different models (read more here).
Dynamic routing to different upstreams and models according to identity and request context (read more here).
Exposing the API publicly for anonymous access (read more here).