> ## Documentation Index
> Fetch the complete documentation index at: https://docs.poolside.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Use Bifrost with Poolside model inference

> Deploy Bifrost as an OpenAI-compatible gateway in front of Poolside model inference.

Use Bifrost as an OpenAI-compatible gateway in front of Poolside model inference when you need centralized routing, virtual keys, budgets, rate limits, spend tracking, request logging, cross-provider fallbacks, or a shared endpoint for internal teams.

Bifrost receives OpenAI-compatible requests from clients, reads the `model` field in each request, and routes the request to the matching upstream model endpoint. In this setup, each upstream endpoint is a Poolside model server running in your Kubernetes cluster.

<Note>
  Bifrost is third-party software. You are responsible for Bifrost configuration, access controls, persistence, upgrades, and security hardening. For production deployment guidance, see the [Bifrost Helm documentation](https://docs.getbifrost.ai/deployment-guides/helm).
</Note>

## How it works

Bifrost ships as a single Go binary that serves the API, administration dashboard, and Prometheus `/metrics` endpoint on one port.

This setup includes:

* One or more Poolside model inference services running in Kubernetes.
* A Bifrost StatefulSet running in the same cluster.
* A persistent volume claim that stores Bifrost configuration and request logs in SQLite.
* A Bifrost ingress that exposes the OpenAI-compatible API and administration dashboard on one endpoint.

Clients send requests to Bifrost instead of calling each Poolside model endpoint directly. Bifrost routes each request to the Poolside model service that matches the provider and model name in the request.

SQLite requires no external database and stores Bifrost state for providers, virtual keys, budgets, and request logs. Bifrost also supports PostgreSQL, Redis, and a vector store for semantic caching, but none are required for this setup.

## Watch the deployment walkthrough

<Frame>
  <iframe className="w-full aspect-video rounded-xl" src="https://www.youtube.com/embed/NHf17R_Srro" title="Deploy Bifrost with Poolside model inference" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowFullScreen />
</Frame>

## Prerequisites

* You have deployed Poolside model inference in a Kubernetes cluster. See [Cloud deployment overview](/deployment/cloud/overview).
* You have a valid `kubeconfig` for the cluster.
* You have `helm`, `kubectl`, `curl`, `jq`, and `openssl` installed on the machine where you run the deployment.
* Your cluster has a default storage class or another storage class that supports `ReadWriteOnce` persistent volumes.
* You have a DNS hostname for the Bifrost endpoint.
* You have an ingress controller that can route traffic to Bifrost.
* You know the namespace where Poolside model inference is running. The examples on this page use `<inference-namespace>`.
* Your Poolside model endpoint does not require an upstream API key. If it does, configure a Bifrost provider key instead of using a keyless provider. See the [Bifrost custom provider documentation](https://docs.getbifrost.ai/providers/custom-providers).

## Step 1: Confirm Poolside model inference is running

Check the model pods and services:

```bash theme={null}
kubectl get pods,svc -n <inference-namespace>
```

Confirm each model pod is ready before you deploy Bifrost. In the standard Poolside inference chart, each model deployment and service is named `inference-<model-key>`, where `<model-key>` is the suffix shown in the `kubectl get pods,svc` output.

Retrieve the served model name from the model deployment:

```bash theme={null}
kubectl describe deploy/inference-<model-key> \
  -n <inference-namespace> | grep served-model-name -A1
```

If the model has an ingress, you can query the model endpoint instead:

```bash theme={null}
curl -s http://<model-hostname>/v1/models | jq -r '.data[].id'
```

To find the model hostname, read it from the model ingress:

```bash theme={null}
kubectl get ingress inference-<model-key> \
  -n <inference-namespace> \
  -o jsonpath='{.spec.rules[0].host}'
```

## Step 2: Add the Bifrost Helm repository

Add the repository and retrieve the latest chart metadata:

```bash theme={null}
helm repo add bifrost https://maximhq.github.io/bifrost/helm-charts
helm repo update
```

## Step 3: Create the Bifrost namespace and secret

Create a namespace for Bifrost:

```bash theme={null}
kubectl create namespace bifrost
```

Create the encryption key that Bifrost uses for sensitive data:

```bash theme={null}
kubectl create secret generic bifrost-encryption-key \
  --from-literal=encryption-key="$(openssl rand -base64 32)" \
  -n bifrost
```

<Warning>
  Store the encryption key securely. Do not change it after Bifrost writes encrypted data unless you also migrate or clear the existing database.
</Warning>

## Step 4: Configure Bifrost Helm values

Create a `values.yaml` file:

```yaml title="Example: values.yaml" theme={null}
image:
  repository: docker.io/maximhq/bifrost
  # The chart requires an explicit image version.
  tag: "<bifrost-image-tag>"

storage:
  mode: sqlite
  persistence:
    enabled: true
    accessMode: ReadWriteOnce
    size: 10Gi
    # storageClass: "<storage-class-name>"

bifrost:
  logLevel: info
  client:
    # Require a virtual key on inference endpoints.
    enforceAuthOnInference: true
    logRetentionDays: 365
  encryptionKeySecret:
    name: bifrost-encryption-key
    key: encryption-key

# Expose the API and administration dashboard through one ingress.
ingress:
  enabled: true
  className: "<ingress-class-name>"
  # These annotations apply to NGINX ingress. Adjust them for your controller.
  annotations:
    # Allow large prompt bodies to avoid HTTP 413 responses.
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
  hosts:
    - host: "<bifrost-hostname>"
      paths:
        - path: /
          pathType: Prefix

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 2000m
    memory: 2Gi
```

Replace:

* `<bifrost-image-tag>` with an available version from [Bifrost images on Docker Hub](https://hub.docker.com/r/maximhq/bifrost/tags). For Red Hat base image requirements, use the `-ubi9` variant of the selected tag.
* `<ingress-class-name>` with the ingress class for your cluster, such as `nginx`.
* `<bifrost-hostname>` with the DNS hostname that routes to Bifrost.
* `<storage-class-name>` with the storage class for the SQLite persistent volume if your cluster does not have a suitable default.

The Bifrost Helm chart sets `bifrost.client.enforceAuthOnInference` to `true`, but the raw Bifrost binary leaves inference authentication disabled. A container that you run locally can therefore behave differently from this Helm deployment. Inference authentication protects `/v1` endpoints with a virtual key.

Dashboard authentication is separate. The chart sets `bifrost.authConfig.isEnabled` to `false` by default, so the dashboard is served without a login. To enable dashboard authentication, configure `authConfig.adminUsername` and `authConfig.adminPassword`, or provide `authConfig.existingSecret`.

<Warning>
  Limit the ingress to a trusted internal network for this initial setup. Before you expose Bifrost outside that network, configure TLS and [Bifrost dashboard authentication](https://docs.getbifrost.ai/deployment-guides/config-json/governance).
</Warning>

## Step 5: Install Bifrost

Install the Bifrost Helm chart:

```bash theme={null}
helm upgrade --install bifrost bifrost/bifrost \
  --namespace bifrost \
  -f values.yaml
```

With SQLite persistence enabled, the chart creates a StatefulSet with one pod and a persistent volume claim. Confirm both are ready:

```bash theme={null}
kubectl get pods,pvc -n bifrost
```

Verify that the gateway and SQLite datastore are healthy:

```bash theme={null}
curl -s http://<bifrost-hostname>/health | jq
```

A healthy response reports an `ok` status for the gateway and database ping.

Before you add a provider, confirm that the gateway starts with no providers configured:

```bash theme={null}
curl -s http://<bifrost-hostname>/api/providers | jq
```

The response reports an empty `providers` array and a `total` of `0`.

## Step 6: Add a Poolside model in Bifrost

1. Gather the served model name from [Step 1](#step-1-confirm-poolside-model-inference-is-running).

2. Find the in-cluster Poolside inference service name:

   ```bash theme={null}
   kubectl get svc -n <inference-namespace>
   ```

3. Open the Bifrost administration dashboard:

   ```text theme={null}
   http://<bifrost-hostname>/
   ```

   With dashboard authentication disabled, the dashboard opens without a login on **Observability > Dashboard**.

4. In the navigation menu, expand **Models** and select **Model Providers**.

5. Select **Add provider**, then select **Custom provider...**.

6. Complete the **Add Custom Provider** form:

   | Bifrost field             | Value                                                                                  |
   | ------------------------- | -------------------------------------------------------------------------------------- |
   | **Name**                  | A short provider name, such as `poolside`. Bifrost uses it as the prefix in model IDs. |
   | **Base Format**           | `OpenAI`                                                                               |
   | **Base URL**              | `http://inference-<model-key>.<inference-namespace>.svc.cluster.local`                 |
   | **Allow Private Network** | On. Bifrost blocks private network addresses by default.                               |
   | **Is Keyless?**           | On for a Poolside model endpoint that does not require an upstream API key.            |
   | **Allowed Request Types** | Leave the defaults, or restrict the provider to the request types your clients need.   |

   Do not append `/v1` to the **Base URL**. The OpenAI base format adds the request path.

7. Select **Add**.

The provider appears on the **Model Providers** page with a **CUSTOM** tag. For a keyless provider, the details confirm that no upstream API keys are required. Its models use namespaced IDs in the form `<provider-name>/<served-model-name>`.

Confirm that Bifrost reports the provider as healthy:

```bash theme={null}
curl -s http://<bifrost-hostname>/api/providers
```

The response lists the provider with a `success` status.

## Step 7: Create a Bifrost virtual key

The Helm chart requires a virtual key for inference requests by default.

Before you create a virtual key, confirm that Bifrost rejects an unauthenticated inference request:

```bash theme={null}
curl -s http://<bifrost-hostname>/v1/models | jq
```

The response body reports `"status_code": 401` and `"type": "virtual_key_required"`.

The **Virtual Keys** page lists existing keys with their assignments, budgets, rate limits, and status.

1. In the navigation menu, expand **Governance** and select **Virtual Keys**.

2. Select **Add Virtual Key**.

3. Complete the form:

   * **Name**: Enter a name that identifies the key in the dashboard and request logs.
   * **Expiry**: Choose a preset from 30 minutes through 7 days, or set an explicit expiration date that matches your organization's key rotation policy.
   * **Provider Configurations**: Select the Poolside provider you created in [Step 6](#step-6-add-a-poolside-model-in-bifrost).
   * **Budget Configuration**: Optionally set a spend limit and reset period.
   * **Rate Limiting Configuration**: Optionally set token and request limits with reset periods from every minute through monthly.

4. Select **Create** and copy the generated key. Bifrost virtual keys start with `sk-bf-`, so OpenAI-compatible clients accept them as API keys without modification.

Store the key securely.

## Step 8: Test the Bifrost endpoint

List the models Bifrost exposes:

```bash theme={null}
curl -s http://<bifrost-hostname>/v1/models \
  -H "Authorization: Bearer <bifrost-virtual-key>" | jq
```

The response includes the namespaced model ID:

```json theme={null}
{
  "data": [
    {
      "id": "<provider-name>/<served-model-name>"
    }
  ]
}
```

Send a chat completion request through Bifrost:

```bash theme={null}
curl -s http://<bifrost-hostname>/v1/chat/completions \
  -H "Authorization: Bearer <bifrost-virtual-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<provider-name>/<served-model-name>",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France? Answer in one short sentence."
      }
    ],
    "max_tokens": 512,
    "temperature": 0
  }'
```

The setup is working when Bifrost returns a chat completion response from the Poolside model.

## Next steps

### Use Bifrost with Poolside Agent CLI

Follow [Install Poolside Agent CLI](/cli/install#install-and-authenticate) and choose **Connect an OpenAI-compatible provider**. Enter:

* **API base URL**: `http://<bifrost-hostname>/v1`
* **API key**: The Bifrost virtual key from [Step 7](#step-7-create-a-bifrost-virtual-key)

Bifrost lists the namespaced model IDs at `/v1/models`. Start `pool`, then press `Ctrl+M` or use `/model` to choose a model. See [Change the agent](/cli/interactive-mode#change-the-agent).

### Use the Bifrost endpoint from other clients

Point any OpenAI-compatible client, such as the OpenAI SDK, at Bifrost. Enter:

* **API base URL**: `http://<bifrost-hostname>/v1`
* **API key**: A Bifrost virtual key

Set the request's `model` value to a namespaced model ID from Bifrost.

## Operational considerations

* **Authentication**: Create separate virtual keys for users, teams, and applications. Apply provider restrictions, budgets, rate limits, and expiration policies instead of sharing one key.
* **Observability**: Scrape the Prometheus `/metrics` endpoint to monitor Bifrost.
* **Persistence**: Bifrost stores configuration and request logs in SQLite on the persistent volume claim. Back up the data according to your organization's recovery requirements.
* **High availability**: Use PostgreSQL and Bifrost's production deployment guidance before you scale the gateway beyond one replica.
* **Network access**: Bifrost must be able to reach the Poolside model service URLs from inside the cluster. Custom providers need **Allow Private Network** enabled for in-cluster service addresses.
* **TLS**: Expose Bifrost over HTTPS before sharing the endpoint outside a trusted internal network.
* **Model names**: Clients use Bifrost's namespaced `<provider-name>/<served-model-name>` ID. The served model name must match the model name returned by the Poolside model endpoint.

## Related resources

* [Cloud deployment overview](/deployment/cloud/overview)
* [Install on Kubernetes](/deployment/cloud/upstream-kubernetes/install)
* [Poolside API](/api/overview)
* [Bifrost Helm documentation](https://docs.getbifrost.ai/deployment-guides/helm)
* [Bifrost custom provider documentation](https://docs.getbifrost.ai/providers/custom-providers)
* [Bifrost virtual key documentation](https://docs.getbifrost.ai/features/governance/virtual-keys)