Use Bifrost with Poolside model inference

Use Bifrost as an OpenAI-compatible gateway in front of Poolside model inference when you need centralized routing, virtual keys, budgets, rate limits, spend tracking, request logging, cross-provider fallbacks, or a shared endpoint for internal teams. Bifrost receives OpenAI-compatible requests from clients, reads the model field in each request, and routes the request to the matching upstream model endpoint. In this setup, each upstream endpoint is a Poolside model server running in your Kubernetes cluster.

Bifrost is third-party software. You are responsible for Bifrost configuration, access controls, persistence, upgrades, and security hardening. For production deployment guidance, see the Bifrost Helm documentation.

How it works

Bifrost ships as a single Go binary that serves the API, administration dashboard, and Prometheus /metrics endpoint on one port. This setup includes:

One or more Poolside model inference services running in Kubernetes.
A Bifrost StatefulSet running in the same cluster.
A persistent volume claim that stores Bifrost configuration and request logs in SQLite.
A Bifrost ingress that exposes the OpenAI-compatible API and administration dashboard on one endpoint.

Clients send requests to Bifrost instead of calling each Poolside model endpoint directly. Bifrost routes each request to the Poolside model service that matches the provider and model name in the request. SQLite requires no external database and stores Bifrost state for providers, virtual keys, budgets, and request logs. Bifrost also supports PostgreSQL, Redis, and a vector store for semantic caching, but none are required for this setup.

Watch the deployment walkthrough

Prerequisites

You have deployed Poolside model inference in a Kubernetes cluster. See Cloud deployment overview.
You have a valid kubeconfig for the cluster.
You have helm, kubectl, curl, jq, and openssl installed on the machine where you run the deployment.
Your cluster has a default storage class or another storage class that supports ReadWriteOnce persistent volumes.
You have a DNS hostname for the Bifrost endpoint.
You have an ingress controller that can route traffic to Bifrost.
You know the namespace where Poolside model inference is running. The examples on this page use <inference-namespace>.
Your Poolside model endpoint does not require an upstream API key. If it does, configure a Bifrost provider key instead of using a keyless provider. See the Bifrost custom provider documentation.

Step 1: Confirm Poolside model inference is running

Check the model pods and services:

kubectl get pods,svc -n <inference-namespace>

Confirm each model pod is ready before you deploy Bifrost. In the standard Poolside inference chart, each model deployment and service is named inference-<model-key>, where <model-key> is the suffix shown in the kubectl get pods,svc output. Retrieve the served model name from the model deployment:

kubectl describe deploy/inference-<model-key> \
  -n <inference-namespace> | grep served-model-name -A1

If the model has an ingress, you can query the model endpoint instead:

curl -s http://<model-hostname>/v1/models | jq -r '.data[].id'

To find the model hostname, read it from the model ingress:

kubectl get ingress inference-<model-key> \
  -n <inference-namespace> \
  -o jsonpath='{.spec.rules[0].host}'

Step 2: Add the Bifrost Helm repository

Add the repository and retrieve the latest chart metadata:

helm repo add bifrost https://maximhq.github.io/bifrost/helm-charts
helm repo update

Step 3: Create the Bifrost namespace and secret

Create a namespace for Bifrost:

kubectl create namespace bifrost

Create the encryption key that Bifrost uses for sensitive data:

kubectl create secret generic bifrost-encryption-key \
  --from-literal=encryption-key="$(openssl rand -base64 32)" \
  -n bifrost

Store the encryption key securely. Do not change it after Bifrost writes encrypted data unless you also migrate or clear the existing database.

Step 4: Configure Bifrost Helm values

Create a values.yaml file:

Example: values.yaml

image:
  repository: docker.io/maximhq/bifrost
  # The chart requires an explicit image version.
  tag: "<bifrost-image-tag>"

storage:
  mode: sqlite
  persistence:
    enabled: true
    accessMode: ReadWriteOnce
    size: 10Gi
    # storageClass: "<storage-class-name>"

bifrost:
  logLevel: info
  client:
    # Require a virtual key on inference endpoints.
    enforceAuthOnInference: true
    logRetentionDays: 365
  encryptionKeySecret:
    name: bifrost-encryption-key
    key: encryption-key

# Expose the API and administration dashboard through one ingress.
ingress:
  enabled: true
  className: "<ingress-class-name>"
  # These annotations apply to NGINX ingress. Adjust them for your controller.
  annotations:
    # Allow large prompt bodies to avoid HTTP 413 responses.
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
  hosts:
    - host: "<bifrost-hostname>"
      paths:
        - path: /
          pathType: Prefix

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2000m
    memory: 2Gi

Replace:

<bifrost-image-tag> with an available version from Bifrost images on Docker Hub. For Red Hat base image requirements, use the -ubi9 variant of the selected tag.
<ingress-class-name> with the ingress class for your cluster, such as nginx.
<bifrost-hostname> with the DNS hostname that routes to Bifrost.
<storage-class-name> with the storage class for the SQLite persistent volume if your cluster does not have a suitable default.

The Bifrost Helm chart sets bifrost.client.enforceAuthOnInference to true, but the raw Bifrost binary leaves inference authentication disabled. A container that you run locally can therefore behave differently from this Helm deployment. Inference authentication protects /v1 endpoints with a virtual key. Dashboard authentication is separate. The chart sets bifrost.authConfig.isEnabled to false by default, so the dashboard is served without a login. To enable dashboard authentication, configure authConfig.adminUsername and authConfig.adminPassword, or provide authConfig.existingSecret.

Limit the ingress to a trusted internal network for this initial setup. Before you expose Bifrost outside that network, configure TLS and Bifrost dashboard authentication.

Step 5: Install Bifrost

Install the Bifrost Helm chart:

helm upgrade --install bifrost bifrost/bifrost \
  --namespace bifrost \
  -f values.yaml

With SQLite persistence enabled, the chart creates a StatefulSet with one pod and a persistent volume claim. Confirm both are ready:

kubectl get pods,pvc -n bifrost

Verify that the gateway and SQLite datastore are healthy:

curl -s http://<bifrost-hostname>/health | jq

A healthy response reports an ok status for the gateway and database ping. Before you add a provider, confirm that the gateway starts with no providers configured:

curl -s http://<bifrost-hostname>/api/providers | jq

The response reports an empty providers array and a total of 0.

Step 6: Add a Poolside model in Bifrost

Gather the served model name from Step 1.
Find the in-cluster Poolside inference service name:
kubectl get svc -n <inference-namespace>
Open the Bifrost administration dashboard:
http://<bifrost-hostname>/
With dashboard authentication disabled, the dashboard opens without a login on Observability > Dashboard.
In the navigation menu, expand Models and select Model Providers.
Select Add provider, then select Custom provider….

Complete the Add Custom Provider form:

Bifrost field	Value
Name	A short provider name, such as `poolside`. Bifrost uses it as the prefix in model IDs.
Base Format	`OpenAI`
Base URL	`http://inference-<model-key>.<inference-namespace>.svc.cluster.local`
Allow Private Network	On. Bifrost blocks private network addresses by default.
Is Keyless?	On for a Poolside model endpoint that does not require an upstream API key.
Allowed Request Types	Leave the defaults, or restrict the provider to the request types your clients need.

Do not append /v1 to the Base URL. The OpenAI base format adds the request path.

Select Add.

The provider appears on the Model Providers page with a CUSTOM tag. For a keyless provider, the details confirm that no upstream API keys are required. Its models use namespaced IDs in the form <provider-name>/<served-model-name>. Confirm that Bifrost reports the provider as healthy:

curl -s http://<bifrost-hostname>/api/providers

The response lists the provider with a success status.

Step 7: Create a Bifrost virtual key

The Helm chart requires a virtual key for inference requests by default. Before you create a virtual key, confirm that Bifrost rejects an unauthenticated inference request:

curl -s http://<bifrost-hostname>/v1/models | jq

The response body reports "status_code": 401 and "type": "virtual_key_required". The Virtual Keys page lists existing keys with their assignments, budgets, rate limits, and status.

In the navigation menu, expand Governance and select Virtual Keys.
Select Add Virtual Key.
Complete the form:
- Name: Enter a name that identifies the key in the dashboard and request logs.
- Expiry: Choose a preset from 30 minutes through 7 days, or set an explicit expiration date that matches your organization’s key rotation policy.
- Provider Configurations: Select the Poolside provider you created in Step 6.
- Budget Configuration: Optionally set a spend limit and reset period.
- Rate Limiting Configuration: Optionally set token and request limits with reset periods from every minute through monthly.
Select Create and copy the generated key. Bifrost virtual keys start with sk-bf-, so OpenAI-compatible clients accept them as API keys without modification.

Store the key securely.

Step 8: Test the Bifrost endpoint

List the models Bifrost exposes:

curl -s http://<bifrost-hostname>/v1/models \
  -H "Authorization: Bearer <bifrost-virtual-key>" | jq

The response includes the namespaced model ID:

{
  "data": [
    {
      "id": "<provider-name>/<served-model-name>"
    }
  ]
}

Send a chat completion request through Bifrost:

curl -s http://<bifrost-hostname>/v1/chat/completions \
  -H "Authorization: Bearer <bifrost-virtual-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<provider-name>/<served-model-name>",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France? Answer in one short sentence."
      }
    ],
    "max_tokens": 512,
    "temperature": 0
  }'

The setup is working when Bifrost returns a chat completion response from the Poolside model.

Next steps

Use Bifrost with Poolside Agent CLI

Follow Install Poolside Agent CLI and choose Connect an OpenAI-compatible provider. Enter:

API base URL: http://<bifrost-hostname>/v1
API key: The Bifrost virtual key from Step 7

Bifrost lists the namespaced model IDs at /v1/models. Start pool, then press Ctrl+M or use /model to choose a model. See Change the agent.

Use the Bifrost endpoint from other clients

Point any OpenAI-compatible client, such as the OpenAI SDK, at Bifrost. Enter:

API base URL: http://<bifrost-hostname>/v1
API key: A Bifrost virtual key

Set the request’s model value to a namespaced model ID from Bifrost.

Operational considerations

Authentication: Create separate virtual keys for users, teams, and applications. Apply provider restrictions, budgets, rate limits, and expiration policies instead of sharing one key.
Observability: Scrape the Prometheus /metrics endpoint to monitor Bifrost.
Persistence: Bifrost stores configuration and request logs in SQLite on the persistent volume claim. Back up the data according to your organization’s recovery requirements.
High availability: Use PostgreSQL and Bifrost’s production deployment guidance before you scale the gateway beyond one replica.
Network access: Bifrost must be able to reach the Poolside model service URLs from inside the cluster. Custom providers need Allow Private Network enabled for in-cluster service addresses.
TLS: Expose Bifrost over HTTPS before sharing the endpoint outside a trusted internal network.
Model names: Clients use Bifrost’s namespaced <provider-name>/<served-model-name> ID. The served model name must match the model name returned by the Poolside model endpoint.

​How it works

​Watch the deployment walkthrough

​Prerequisites

​Step 1: Confirm Poolside model inference is running

​Step 2: Add the Bifrost Helm repository

​Step 3: Create the Bifrost namespace and secret

​Step 4: Configure Bifrost Helm values

​Step 5: Install Bifrost

​Step 6: Add a Poolside model in Bifrost

​Step 7: Create a Bifrost virtual key

​Step 8: Test the Bifrost endpoint

​Next steps

​Use Bifrost with Poolside Agent CLI

​Use the Bifrost endpoint from other clients

​Operational considerations

​Related resources

How it works

Watch the deployment walkthrough

Prerequisites

Step 1: Confirm Poolside model inference is running

Step 2: Add the Bifrost Helm repository

Step 3: Create the Bifrost namespace and secret

Step 4: Configure Bifrost Helm values

Step 5: Install Bifrost

Step 6: Add a Poolside model in Bifrost

Step 7: Create a Bifrost virtual key

Step 8: Test the Bifrost endpoint

Next steps

Use Bifrost with Poolside Agent CLI

Use the Bifrost endpoint from other clients

Operational considerations

Related resources