Install on-premises

Overview

Use this guide to deploy Poolside model inference on a dedicated GPU workstation or server host. The on-premises installation bundle can be used in internet-connected or air-gapped environments. When the bundle is already cached on the host, installation typically takes about one hour. The installation process has the following phases:

Install RKE2 infrastructure.
Install supporting infrastructure services.
Upload model checkpoints.
Deploy model inference and ingress.

The installation includes:

RKE2 Kubernetes
S3-compatible object storage for model checkpoints
A local container registry
cert-manager for self-signed certificates
NVIDIA GPU Operator for GPU access in RKE2 workloads
Model inference workloads for the model checkpoints you provide

Model checkpoint files are provided separately based on your deployment. Upload the model checkpoint files during the model upload step.

Prerequisites

Before you begin, complete the on-premises installation prerequisites, including the preparation steps for your host operating system. After you complete the prerequisites for your operating system, every host follows the same installation sequence on this page. In an air-gapped environment, complete Step 0 first and use the air-gapped Terraform commands in Steps 1 through 4. Use the Poolside installation bundle that matches your host architecture, either linux/amd64 or linux/arm64.

Step 0 (optional): Set up an air-gapped installation

This configuration is required for air-gapped installations. In internet-connected environments, you can skip this step.

To use the local Terraform provider cache included in the bundle, configure Terraform to load providers from the bundled terraform.d directory.

Locate poolside-terraform.tfrc in the root of the unpacked installation bundle.
Replace the $POOLSIDE_INSTALL_DIR placeholder with the fully qualified path to the bundle’s root directory.
For Terraform commands in the installation steps, prefix the command with the Terraform CLI configuration path:
TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc terraform <command>

Setting this variable ensures that both root and non-root users reference the same cached Terraform providers.

You can configure Terraform using alternative methods, such as a .terraformrc file, as described in the official HashiCorp documentation. Because the installation process runs as both root and a local user, you must ensure that both accounts are configured to reference the cached providers correctly.

Step 1: Install RKE2 on the host

The 01-infra-rke2 directory contains the Terraform module that installs RKE2 on the host. Using sudo, run the following commands from the 01-infra-rke2 directory.

You must run the RKE2 installation using sudo from the same user account that runs Poolside model inference after deployment.Terraform uses the original user and group IDs from the sudo environment to set ownership and permissions required by later installation stages.

Air-gapped environment:

sudo TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc /usr/local/bin/terraform init
sudo TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc /usr/local/bin/terraform apply

Internet-connected environment:

sudo /usr/local/bin/terraform init
sudo /usr/local/bin/terraform apply

If RKE2 certificates or credentials change, re-run this step to refresh the configuration files that restore access for the installation user.

This step exports a kubeconfig readable by the installing user to <bundle-path>/poolside-install/.kube/config. Use this file for any kubectl commands during and after installation by exporting it into your shell:

export KUBECONFIG=<bundle-path>/poolside-install/.kube/config

The RKE2-managed original at /etc/rancher/rke2/rke2.yaml is readable by root only.

Step 2: Install supporting infrastructure services

The 02-infra-services directory contains the Terraform module that accesses the RKE2 cluster and deploys the supporting infrastructure required by Poolside model inference. This step installs:

A local container registry
S3-compatible object storage
Ingress and certificate resources for inference endpoints
NVIDIA GPU Operator, deployed as gpu-operator

Before you run Terraform, complete the following configuration steps.

1. Configure ingress hostnames

In 02-infra-services/terraform.tfvars, set poolside_ingress_hosts to the list of model hostname prefixes that you plan to serve (one per model). Each prefix is combined with the domain_name variable (default poolside.local) to form the fully qualified domain name (FQDN) that the model is served from. The installer uses these FQDNs when it creates self-signed certificate SANs. For example, the prefix poolside-models-agent with the default domain_name produces the FQDN poolside-models-agent.poolside.local:

02-infra-services/terraform.tfvars

poolside_ingress_hosts = [
  "poolside-models-agent"
]
# domain_name defaults to "poolside.local", so the model is served from
# https://poolside-models-agent.poolside.local

The FQDN formed here must match the ingress_host_name that you later provide for the model in Step 4: Deploy Poolside model inference, and it is the hostname you add to DNS (or /etc/hosts) after installation. If you use installer-generated self-signed certificates, each model ingress_host_name that you configure in Step 4 must match one of the FQDNs formed from poolside_ingress_hosts. This lets the installer generate certificates with the required SANs before model inference is deployed. If you use custom TLS certificates, ensure that your certificate SANs include each model ingress_host_name that you configure in Step 4. The installer includes poolside-docs in certificate SANs by default. Add a documentation hostname prefix to poolside_ingress_hosts only if you want to use a different documentation hostname.

2. Configure custom TLS certificates

Skip this step if you use installer-generated self-signed certificates. If you use custom TLS certificates, you must provide your own CA and server certificate before you run terraform apply. The custom_certificates and custom_ca_trust_chain parameters configure certificates for the TLS-terminating inference and storage services. The custom_certificates schema accepts certificate and key entries for poolside, services.storage, and services.storage_s3. You can use one certificate that covers all exposed hostnames, or separate certificates if your Public Key Infrastructure (PKI) requires it. Across all certificates you provide, the Subject Alternative Names (SANs) must cover every hostname that you expose, including:

Every model ingress_host_name that you configure in 04-poolside-inference/terraform.tfvars
Any custom documentation ingress hostname that you configure instead of the default poolside-docs hostname
seaweedfs.poolside.local
seaweedfs-s3.poolside.local

Place your CA certificate, server certificate, and private key in a directory accessible to Terraform. The example below uses <bundle-path>/poolside-install/byo-certs/. The poolside-install/ subdirectory holds the installation’s persistent state and is preserved across cluster resets, so it is the recommended location for BYO certificate files.
<bundle-path>/poolside-install/byo-certs/ ├── ca.crt # CA certificate (root, or root and intermediate chain) ├── server.crt # Server certificate signed by the CA └── server.key # Server private key
You must reference these files using fully qualified (absolute) paths in the next step. Relative paths are not supported.

In 02-infra-services/terraform.tfvars, set the BYO variables:

custom_ca_trust_chain = {
  root_ca_path = "<bundle-path>/poolside-install/byo-certs/ca.crt"
}

custom_certificates = {
  poolside = {
    cert_path = "<bundle-path>/poolside-install/byo-certs/server.crt"
    key_path  = "<bundle-path>/poolside-install/byo-certs/server.key"
  }
  services = {
    storage = {
      cert_path = "<bundle-path>/poolside-install/byo-certs/server.crt"
      key_path  = "<bundle-path>/poolside-install/byo-certs/server.key"
    }
    storage_s3 = {
      cert_path = "<bundle-path>/poolside-install/byo-certs/server.crt"
      key_path  = "<bundle-path>/poolside-install/byo-certs/server.key"
    }
  }
}

custom_ca_trust_chain.root_ca_path must point to the CA that signed server.crt. When you run terraform apply, the module creates Kubernetes secrets with a -byo suffix from these files.

3. Configure an internal APT mirror on Ubuntu (optional)

If the GPU Operator on Ubuntu uses an internal APT mirror, complete Use an internal APT mirror before you run Terraform. Skip this step for other GPU driver installation methods.

4. Run Terraform

Using sudo, run the following commands from the 02-infra-services directory.

You must run this step using sudo from the same user account that runs Poolside model inference after deployment.Terraform uses the original user and group IDs from the sudo environment to set the permissions required for RKE2 cluster access in later stages.

Air-gapped environment:

sudo TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc /usr/local/bin/terraform init
sudo TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc /usr/local/bin/terraform apply

Internet-connected environment:

sudo /usr/local/bin/terraform init
sudo /usr/local/bin/terraform apply

This step can take some time to complete. The process loads container images into the local RKE2 registry to support disconnected operation and improve Poolside startup performance.

Step 3: Upload Poolside models

The 03-poolside-model-upload directory contains the Terraform module that uploads model checkpoints into the deployment’s S3-compatible storage. The module creates a Kubernetes job that syncs model files from a local host directory into the poolside-models bucket.

Unpack the Poolside model checkpoint into the local host directory /opt/poolside/poolside-model-uploads. The checkpoint is shipped as a tar archive. Extract it so that the checkpoint directory sits directly under the volume. For example:
tar -xf <model-checkpoint-name>.tar -C /opt/poolside/poolside-model-uploads
The resulting directory name becomes the model’s S3 path. In this example, the checkpoint is uploaded to s3://poolside-models/<model-checkpoint-name>, which is the s3_uri value that you reference in Step 4: Deploy Poolside model inference. /opt/poolside/poolside-model-uploads is the default location. If you customized the Poolside host volume location in 01-infra-rke2, use the corresponding directory instead.

Run the following commands from the 03-poolside-model-upload directory. Air-gapped environment:

TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc terraform init
TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc terraform apply

Internet-connected environment:

terraform init
terraform apply

To upload additional or updated models later, repeat these steps. Uploads are additive and do not remove existing models from the deployment.

The apply creates a Kubernetes job that syncs the volume into the poolside-models bucket and then waits for that job to complete, so the apply runs for as long as the upload itself. A checkpoint larger than 30 GB can take several minutes to upload from local disk. Terraform waits up to 20 minutes by default. To raise this limit for larger models or slower storage, set overrides.upload_timeout in 03-poolside-model-upload/terraform.tfvars. If the apply times out with an error of the form job: poolside-services/poolside-model-upload-... is not in complete state, the upload job usually finishes shortly afterward. Wait for the job to report Complete, then re-run terraform apply. The re-run re-syncs incrementally and completes quickly.

export KUBECONFIG=<bundle-path>/poolside-install/.kube/config
kubectl get jobs -n poolside-services

Step 4: Deploy Poolside model inference

The 04-poolside-inference directory contains the Terraform module that deploys the inference containers used to serve Poolside models.

In the 04-poolside-inference directory, update terraform.tfvars with the model details you want to deploy. The key for each entry in the models map is the model ID that clients use in API requests. Set s3_uri to the checkpoint directory uploaded in Step 3. Set ingress_host_name to the FQDN formed from a poolside_ingress_hosts prefix and domain_name in Step 2: Install supporting infrastructure services. If you use custom TLS certificates, the certificate SANs must include each model ingress_host_name.
Example: Model configuration
deployment_name = "poolside-server" models = { agent = { s3_uri = "s3://poolside-models/<model-checkpoint-name>" ingress_host_name = "poolside-models-agent.poolside.local" gpus = 2 replicas = 1 model_type = "agent" } }

Run the following commands from the 04-poolside-inference directory. Air-gapped environment:

TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc terraform init
TF_CLI_CONFIG_FILE=<bundle-path>/poolside-terraform.tfrc terraform apply

Internet-connected environment:

terraform init
terraform apply

Next steps: Post-installation configuration

Configure local DNS

Add hostname resolution on the deployment host. Replace the model ingress hostnames with the ingress_host_name values you configured in Step 4. If you expose multiple model ingress hostnames, include each hostname on the same line.

cat <<EOF | sudo tee -a /etc/hosts
127.0.0.1 <model-ingress-host> <additional-model-ingress-host> seaweedfs.poolside.local seaweedfs-s3.poolside.local
EOF

For example:

cat <<EOF | sudo tee -a /etc/hosts
127.0.0.1 poolside-models-agent.poolside.local seaweedfs.poolside.local seaweedfs-s3.poolside.local
EOF

Validate the deployment

The kubectl commands in this section use the kubeconfig that Step 1: Install RKE2 on the host exported for the installing user. Export it into your shell before you run them:

export KUBECONFIG=<bundle-path>/poolside-install/.kube/config

Expected cluster state

Model startup takes several minutes after the Step 4 apply completes. An init container pulls the checkpoint from S3-compatible storage, then vLLM loads the weights onto the GPUs. Watch for the model pod to show READY 1/1 and STATUS Running:

kubectl get pods -n poolside-models -w

A healthy installation has all pods Running or Completed across the kube-system, gpu-operator, poolside-cert-manager, poolside-registry, poolside-services, and poolside-models namespaces:

kubectl get pods -A

Confirm that the model inference endpoint resolves to the deployment host:

getent hosts <model-ingress-host>

Confirm that the model upload job completed successfully. The upload job runs in the poolside-services namespace:

kubectl get jobs -n poolside-services

You can confirm GPU usage with nvidia-smi on the host once the model pod is ready.

Trust the deployment TLS certificate

The installation host already trusts the deployment’s certificates. Step 2: Install supporting infrastructure services installs the self-signed CA into the host operating system’s trust store, so curl and other clients running on the host verify TLS with no extra flags. On Ubuntu, the CA is under /usr/local/share/ca-certificates/poolside/. The location varies by operating system. External clients, such as developer workstations and CI systems, need the CA certificate to complete a trusted TLS handshake. Step 2 exports the CA to <bundle-path>/poolside-install/certs/poolside-ca.crt. Copy that file to the external client and either install it into the client’s trust store or pass it to curl with --cacert. If you provided your own externally valid certificates, use your own CA instead.

Send a first request

Poolside models are served over an OpenAI-compatible API at https://<model-ingress-host>/v1. The model ID is the key of the models map from Step 4: Deploy Poolside model inference, such as agent in the worked example. The endpoint URL is also available from terraform output in the 04-poolside-inference directory. The commands below run on the installation host, which already trusts the deployment CA. From an external client, add --cacert <ca-certificate-path> as described above. List the available models:

curl https://poolside-models-agent.poolside.local/v1/models

Send a first chat completion:

curl -X POST https://poolside-models-agent.poolside.local/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "agent",
    "messages": [{"role": "user", "content": "Write a one-line Bash command to count files in a directory."}],
    "max_tokens": 200
  }'

A JSON response with a chat.completion object confirms that the deployment serves inference end to end.

Troubleshooting

The kubectl commands in this section use the kubeconfig exported in Step 1: Install RKE2 on the host. If you have not already done so in your current shell, export it first:

export KUBECONFIG=<bundle-path>/poolside-install/.kube/config

Model pods stuck in `ContainerCreating`

Confirm that the host detects NVIDIA GPU devices:
lspci | grep -i nvidia

Confirm that Kubernetes reports GPUs as allocatable:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

Check model workload status:
kubectl get pods -n poolside-models

Models not loading

Confirm that the model checkpoint files were copied into /opt/poolside/poolside-model-uploads, or into the custom host volume location you configured.
Confirm that the model upload job completed successfully.

Check model initialization logs:

kubectl logs <pod-name> -c model-downloader -n poolside-models

Useful commands

# Check overall cluster status.
kubectl get pods -A

# Monitor model workloads.
kubectl get pods -n poolside-models

# Monitor supporting services.
kubectl get pods -n poolside-services

# Inspect pod and deployment details.
kubectl describe pod <pod-name> -n <namespace>
kubectl describe deploy <deployment-name> -n <namespace>

# View logs.
kubectl logs <pod-name> -n <namespace>

# View recent events in a namespace.
kubectl get events -n <namespace>

​Overview

​Prerequisites

​Step 0 (optional): Set up an air-gapped installation

​Step 1: Install RKE2 on the host

​Step 2: Install supporting infrastructure services

​1. Configure ingress hostnames

​2. Configure custom TLS certificates

​3. Configure an internal APT mirror on Ubuntu (optional)

​4. Run Terraform

​Step 3: Upload Poolside models

​Step 4: Deploy Poolside model inference

​Next steps: Post-installation configuration

​Configure local DNS

​Validate the deployment

​Expected cluster state

​Trust the deployment TLS certificate

​Send a first request

​Troubleshooting

​Model pods stuck in ContainerCreating

​Models not loading

​Useful commands

​Related resources

Overview

Prerequisites

Step 0 (optional): Set up an air-gapped installation

Step 1: Install RKE2 on the host

Step 2: Install supporting infrastructure services

1. Configure ingress hostnames

2. Configure custom TLS certificates

3. Configure an internal APT mirror on Ubuntu (optional)

4. Run Terraform

Step 3: Upload Poolside models

Step 4: Deploy Poolside model inference

Next steps: Post-installation configuration

Configure local DNS

Validate the deployment

Expected cluster state

Trust the deployment TLS certificate

Send a first request

Troubleshooting

Model pods stuck in `ContainerCreating`

Models not loading

Useful commands

Related resources