# Self-Hosted Secure Remote Access to llama.cpp > Octelium documentation. Canonical page: . Octelium enables you to seamlessly provide secure access to [llama.cpp](https://github.com/ggml-org/llama.cpp) server via both the private client-based mode over WireGuard/QUIC as well as the public clientless secure access (read more about the clientless access mode [here](https://octelium.com/docs/octelium/latest/management/core/service/clientless.md)) for lightweight open models such as Google's [Gemma 3](https://blog.google/technology/developers/gemma-3/), DeepSeek R1, Meta's Llama, Qwen, etc.... The llama.cpp server is served via the dedicated `LLM` *Service* mode (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md)), which understands the inference API rather than treating it as generic HTTP traffic. This enables you to apply to your self-hosted models the very same identity-aware access control by model and token limits, token quotas, guardrails, managed prompts and inference-specific visibility that you apply to SaaS providers (read more about using Octelium as an AI gateway [here](https://octelium.com/docs/octelium/latest/management/guide/service/ai/ai-gateway.md)). ## Deploy llama.cpp Here is a simple example where you can seamlessly deploy a llama.cpp server as a managed container and serve it as an Octelium *Service* (read more about managed containers [here](https://octelium.com/docs/octelium/latest/management/core/service/managed-containers.md)): ```yaml kind: Service metadata: name: llama spec: port: 8080 mode: LLM isPublic: true config: upstream: container: port: 8080 image: ghcr.io/ggml-org/llama.cpp:server args: - --model-url - https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf - --alias - qwen2.5-0.5b-instruct - --host - 0.0.0.0 - --port - "8080" - -c - "2048" resourceLimit: cpu: millicores: 2000 memory: megabytes: 3000 llm: protocol: OPENAI ``` The llama.cpp server serves the OpenAI-compatible API over the canonical `/v1` routes, so we use the `OPENAI` protocol (read more about the supported protocols [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#protocols)). The `--alias` flag sets a stable model name that is reported by the server and that you can use in your *Policies* regardless of the underlying GGUF file name. Note that the server must listen on `0.0.0.0` in order to be reachable. If your underlying Kubernetes installation supports requesting and scheduling GPUs (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/managed-containers.md#extended-resources)), you can modify the above configuration to be as follows: ```yaml kind: Service metadata: name: llama spec: port: 8080 mode: LLM isPublic: true config: upstream: container: port: 8080 # !mark image: ghcr.io/ggml-org/llama.cpp:server-cuda args: - --model-url - https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf - --alias - qwen2.5-0.5b-instruct - --host - 0.0.0.0 - --port - "8080" - -c - "2048" resourceLimit: cpu: millicores: 2000 memory: megabytes: 3000 #!mark(1:3) ext: # Change this according to your Kubernetes cluster available values nvidia.com/gpu: "1" llm: protocol: OPENAI ``` You can now apply the creation of the *Service* as follows (read more [here](https://octelium.com/docs/octelium/latest/management/core/overview.md)): ```bash octeliumctl apply /PATH/TO/SERVICE.YAML ``` You can also serve a llama.cpp server that's served by a connected *User* (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/overview.md#remotely-via-a-connected-user)), for example from a GPU-equipped machine behind NAT, by simply setting the upstream to `url: http://localhost:8080` and `user: ` instead of the managed container. ## Secretless Access The llama.cpp server can itself require an API key via its `--api-key` flag or its `LLAMA_API_KEY` environment variable. This protects the server even from workloads that can reach it directly from within your network. Octelium can inject that API key on-the-fly (read more about secretless access [here](https://octelium.com/docs/octelium/latest/management/core/service/secretless.md)) so that neither your *Users* nor your applications ever see it. First, we create a *Secret* for the API key (read more [here](https://octelium.com/docs/octelium/latest/management/core/secret.md)): ```bash octeliumctl create secret llama-api-key ``` Now we use the same *Secret* both as the server's API key and as the upstream bearer credential of the *Service* as follows: ```yaml kind: Service metadata: name: llama spec: port: 8080 mode: LLM isPublic: true config: upstream: container: port: 8080 image: ghcr.io/ggml-org/llama.cpp:server args: - --model-url - https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/qwen2.5-0.5b-instruct-q8_0.gguf - --alias - qwen2.5-0.5b-instruct - --host - 0.0.0.0 - --port - "8080" - -c - "2048" # !mark(1:3) env: - name: LLAMA_API_KEY fromSecret: llama-api-key resourceLimit: cpu: millicores: 2000 memory: megabytes: 3000 llm: protocol: OPENAI # !mark(1:3) auth: bearer: fromSecret: llama-api-key ``` The downstream client authenticates to Octelium. Octelium authenticates separately to the llama.cpp server and never exposes the API key to the client. ## Client-Based Access Now you can access the llama.cpp server as an ordinary OpenAI-compatible server either via the client-based mode via `octelium connect` or via the clientless mode via OAuth2 client credentials or directly via bearer access tokens. In the client-based access mode, *Users* can simply access the *Service* at the base URL `http://llama:8080/v1` via any OpenAI-compatible client, SDK, coding agent or chat UI. For example: ```bash curl http://llama:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is Octelium?"}]}' ``` > **Note:** > > The LLM mode only exposes the inference routes of the protocol and rejects every other route before it reaches the upstream. This includes the non-OpenAI routes of the llama.cpp server such as `/completion`, `/tokenize`, `/slots` and its built-in web UI. ## Clientless Access As for the clientless mode, `WORKLOAD` *Users* can access the *Service* via the standard OAuth2 client credentials in order for your workloads that can be written in any programming language to access the *Service* without having to use any special SDKs or have access to external clients. All you need is to create an `OAUTH2` *Credential* as illustrated [here](https://octelium.com/docs/octelium/latest/management/core/credential.md#oauth2-client-credentials). Now, here is an example written in Typescript: ```typescript import OpenAI from "openai"; import { OAuth2Client } from "@badgateway/oauth2-client"; async function main() { const oauth2Client = new OAuth2Client({ server: "https:///", clientId: "spxg-cdyx", clientSecret: "AQpAzNmdEcPIfWYR2l2zLjMJm....", tokenEndpoint: "/oauth2/token", authenticationMethod: "client_secret_post", }); const oauth2Creds = await oauth2Client.clientCredentials(); const client = new OpenAI({ apiKey: oauth2Creds.accessToken, baseURL: "https://llama./v1", }); const chatCompletion = await client.chat.completions.create({ model: "qwen2.5-0.5b-instruct", messages: [ { role: "user", content: "How do I write a Golang HTTP reverse proxy?" }, ], max_tokens: 1024, }); console.log(chatCompletion.choices[0].message.content); } main(); ``` Note that the base URL includes the `/v1` prefix of the canonical OpenAI routes. You can read more about using OAuth2 client credentials in other programming languages [here](https://octelium.com/docs/octelium/latest/user/clientless/oauth2.md). Regarding authentication for `HUMAN` *Users* (read more [here](https://octelium.com/docs/octelium/latest/management/core/user.md#types)), you can seamlessly authenticate to the *Cluster* via your web browser using an *IdentityProvider* (read more [here](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md)). There are currently 3 types: - GitHub OAuth *IdentityProvider* as shown in detail [here](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md#github-oauth2) - OpenID Connect *IdentityProviders* (e.g. Okta, Auth0, etc...) as shown [here](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md#openid-connect). - SAML 2.0 *IdentityProviders* (e.g. Okta, Entra ID, etc...) as shown [here](https://octelium.com/docs/octelium/latest/management/core/identity-providers.md#saml-20). Furthermore, you can register a FIDO2 *Authenticator* (e.g. Yubikeys) in order to directly login later via Passkey (read more [here](https://octelium.com/docs/octelium/latest/management/core/authenticator.md#passkey-login)) without having to use an *IdentityProvider*. ## Access Control and Guardrails The LLM mode exposes the normalized inference request information under `ctx.request.llm` to your *Policies* (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#access-control)). For example, the following *Policy* only allows members of the `ai-users` group to generate output from the served model without tools and with an explicit output maximum: ```yaml authorization: inlinePolicies: - spec: rules: - effect: ALLOW condition: all: of: - match: '"ai-users" in ctx.user.spec.groups' - match: ctx.request.llm.operation == "GENERATE" - match: ctx.request.llm.model == "qwen2.5-0.5b-instruct" - match: ctx.request.llm.hasTools == false - match: ctx.request.llm.maxOutputTokens > 0 - match: ctx.request.llm.maxOutputTokens <= 2048 ``` You can also use guardrails (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#guardrail)) in order to block detected credentials and redact email addresses before the prompt reaches the model, as well as bound the request size and the requested output (read more about limits [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/overview.md#limits)) as follows: ```yaml llm: protocol: OPENAI limits: maxEstimatedInputTokens: 2048 maxOutputTokens: 2048 plugins: - name: protect-input condition: match: ctx.request.llm.operation == "GENERATE" guardrail: leg: REQUEST scopes: - CONTENT - INSTRUCTIONS - TOOL_RESULTS patterns: - secrets: {} action: DENY - type: EMAIL action: REDACT denyMessage: The request contains restricted content ``` ## Visibility Octelium also provides OpenTelemetry-ready, inference-aware visibility and access logging in real time that includes the authenticated identity, the operation, the requested and served model, the provider-reported token usage, the finish reason and the time to first token (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/visibility.md)). You can read more about visibility [here](https://octelium.com/docs/octelium/latest/management/core/visibility.md). Here are a few more features that you might be interested in: - Token-consumption quotas per *User*, per *Session* or per tenant (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#token-rate-limit)). - Managed system instructions, message injection, tool filtering and reasoning controls (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md)). - Semantic caching and semantic routing between different models (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/llm/plugins.md#semantic-cache)). - Dynamic routing to different upstreams and models according to identity and request context (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/dynamic-config.md)). - Exposing the API publicly for anonymous access (read more [here](https://octelium.com/docs/octelium/latest/management/core/service/anonymous-access.md)).