> ## Documentation Index > Fetch the complete documentation index at: https://docs.poolside.ai/llms.txt > Use this file to discover all available pages before exploring further. # Install on-premises > Install Poolside model inference on a single on-premises GPU host by using the Terraform installation bundle. ## Overview Use this guide to deploy Poolside model inference on a dedicated GPU workstation or server host. The on-premises installation bundle can be used in internet-connected or air-gapped environments. When the bundle is already cached on the host, installation typically takes about one hour. The installation process has the following phases: 1. Install RKE2 infrastructure. 2. Install supporting infrastructure services. 3. Upload model checkpoints. 4. Deploy model inference and ingress. The installation includes: * RKE2 Kubernetes * S3-compatible object storage for model checkpoints * A local container registry * `cert-manager` for self-signed certificates * NVIDIA GPU Operator for GPU access in RKE2 workloads * Model inference workloads for the model checkpoints you provide Model checkpoint files are provided separately based on your deployment. Upload the model checkpoint files during the model upload step. ## Prerequisites Before you begin, complete the [on-premises installation prerequisites](/deployment/on-prem/prerequisites/overview), including the preparation steps for your host operating system. After you complete the prerequisites for your operating system, every host follows the same installation sequence on this page. In an air-gapped environment, complete Step 0 first and use the air-gapped Terraform commands in Steps 1 through 4. Use the Poolside installation bundle that matches your host architecture, either `linux/amd64` or `linux/arm64`. ## Step 0 (optional): Set up an air-gapped installation This configuration is required for air-gapped installations. In internet-connected environments, you can skip this step. To use the local Terraform provider cache included in the bundle, configure Terraform to load providers from the bundled `terraform.d` directory. 1. Locate `poolside-terraform.tfrc` in the root of the unpacked installation bundle. 2. Replace the `$POOLSIDE_INSTALL_DIR` placeholder with the fully qualified path to the bundle's root directory. 3. For Terraform commands in the installation steps, prefix the command with the Terraform CLI configuration path: ```bash theme={null} TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform ``` Setting this variable ensures that both root and non-root users reference the same cached Terraform providers. You can configure Terraform using alternative methods, such as a `.terraformrc` file, as described in the official [HashiCorp documentation](https://developer.hashicorp.com/terraform/cli/v1.8.x/config/config-file). Because the installation process runs as both `root` and a local user, you must ensure that both accounts are configured to reference the cached providers correctly. ## Step 1: Install RKE2 on the host The `01-infra-rke2` directory contains the Terraform module that installs RKE2 on the host. Using `sudo`, run the following commands from the `01-infra-rke2` directory. You must run the RKE2 installation using `sudo` from the same user account that runs Poolside model inference after deployment. Terraform uses the original user and group IDs from the `sudo` environment to set ownership and permissions required by later installation stages. **Air-gapped environment:** ```bash theme={null} sudo TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc /usr/local/bin/terraform init sudo TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc /usr/local/bin/terraform apply ``` **Internet-connected environment:** ```bash theme={null} sudo /usr/local/bin/terraform init sudo /usr/local/bin/terraform apply ``` If RKE2 certificates or credentials change, re-run this step to refresh the configuration files that restore access for the installation user. This step exports a kubeconfig readable by the installing user to `/poolside-install/.kube/config`. Use this file for any `kubectl` commands during and after installation by exporting it into your shell: ```bash theme={null} export KUBECONFIG=/poolside-install/.kube/config ``` The RKE2-managed original at `/etc/rancher/rke2/rke2.yaml` is readable by `root` only. ## Step 2: Install supporting infrastructure services The `02-infra-services` directory contains the Terraform module that accesses the RKE2 cluster and deploys the supporting infrastructure required by Poolside model inference. This step installs: * A local container registry * S3-compatible object storage * Ingress and certificate resources for inference endpoints * NVIDIA GPU Operator, deployed as `gpu-operator` Before you run Terraform, complete the following configuration steps. ### 1. Configure ingress hostnames In `02-infra-services/terraform.tfvars`, set `poolside_ingress_hosts` to the list of model hostname **prefixes** that you plan to serve (one per model). Each prefix is combined with the `domain_name` variable (default `poolside.local`) to form the fully qualified domain name (FQDN) that the model is served from. The installer uses these FQDNs when it creates self-signed certificate SANs. For example, the prefix `poolside-models-agent` with the default `domain_name` produces the FQDN `poolside-models-agent.poolside.local`: ```hcl title="02-infra-services/terraform.tfvars" theme={null} poolside_ingress_hosts = [ "poolside-models-agent" ] # domain_name defaults to "poolside.local", so the model is served from # https://poolside-models-agent.poolside.local ``` The FQDN formed here must match the `ingress_host_name` that you later provide for the model in [Step 4: Deploy Poolside model inference](#step-4-deploy-poolside-model-inference), and it is the hostname you add to DNS (or `/etc/hosts`) after installation. If you use installer-generated self-signed certificates, each model `ingress_host_name` that you configure in Step 4 must match one of the FQDNs formed from `poolside_ingress_hosts`. This lets the installer generate certificates with the required SANs before model inference is deployed. If you use custom TLS certificates, ensure that your certificate SANs include each model `ingress_host_name` that you configure in Step 4. The installer includes `poolside-docs` in certificate SANs by default. Add a documentation hostname prefix to `poolside_ingress_hosts` only if you want to use a different documentation hostname. ### 2. Configure custom TLS certificates Skip this step if you use installer-generated self-signed certificates. If you use custom TLS certificates, you must provide your own CA and server certificate before you run `terraform apply`. The `custom_certificates` and `custom_ca_trust_chain` parameters configure certificates for the TLS-terminating inference and storage services. The `custom_certificates` schema accepts certificate and key entries for `poolside`, `services.storage`, and `services.storage_s3`. You can use one certificate that covers all exposed hostnames, or separate certificates if your Public Key Infrastructure (PKI) requires it. Across all certificates you provide, the Subject Alternative Names (SANs) must cover every hostname that you expose, including: * Every model `ingress_host_name` that you configure in `04-poolside-inference/terraform.tfvars` * Any custom documentation ingress hostname that you configure instead of the default `poolside-docs` hostname * `seaweedfs.poolside.local` * `seaweedfs-s3.poolside.local` 1. Place your CA certificate, server certificate, and private key in a directory accessible to Terraform. The example below uses `/poolside-install/byo-certs/`. The `poolside-install/` subdirectory holds the installation's persistent state and is preserved across cluster resets, so it is the recommended location for BYO certificate files. ```text theme={null} /poolside-install/byo-certs/ ├── ca.crt # CA certificate (root, or root and intermediate chain) ├── server.crt # Server certificate signed by the CA └── server.key # Server private key ``` You must reference these files using fully qualified (absolute) paths in the next step. Relative paths are not supported. 2. In `02-infra-services/terraform.tfvars`, set the BYO variables: ```hcl theme={null} custom_ca_trust_chain = { root_ca_path = "/poolside-install/byo-certs/ca.crt" } custom_certificates = { poolside = { cert_path = "/poolside-install/byo-certs/server.crt" key_path = "/poolside-install/byo-certs/server.key" } services = { storage = { cert_path = "/poolside-install/byo-certs/server.crt" key_path = "/poolside-install/byo-certs/server.key" } storage_s3 = { cert_path = "/poolside-install/byo-certs/server.crt" key_path = "/poolside-install/byo-certs/server.key" } } } ``` `custom_ca_trust_chain.root_ca_path` must point to the CA that signed `server.crt`. When you run `terraform apply`, the module creates Kubernetes secrets with a `-byo` suffix from these files. ### 3. Configure an internal APT mirror on Ubuntu (optional) If the GPU Operator on Ubuntu uses an internal APT mirror, complete [Use an internal APT mirror](/deployment/on-prem/prerequisites/ubuntu#use-an-internal-apt-mirror) before you run Terraform. Skip this step for other GPU driver installation methods. ### 4. Run Terraform Using `sudo`, run the following commands from the `02-infra-services` directory. You must run this step using `sudo` from the same user account that runs Poolside model inference after deployment. Terraform uses the original user and group IDs from the `sudo` environment to set the permissions required for RKE2 cluster access in later stages. **Air-gapped environment:** ```bash theme={null} sudo TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc /usr/local/bin/terraform init sudo TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc /usr/local/bin/terraform apply ``` **Internet-connected environment:** ```bash theme={null} sudo /usr/local/bin/terraform init sudo /usr/local/bin/terraform apply ``` This step can take some time to complete. The process loads container images into the local RKE2 registry to support disconnected operation and improve Poolside startup performance. ## Step 3: Upload Poolside models The `03-poolside-model-upload` directory contains the Terraform module that uploads model checkpoints into the deployment's S3-compatible storage. The module creates a Kubernetes job that syncs model files from a local host directory into the `poolside-models` bucket. 1. Unpack the Poolside model checkpoint into the local host directory `/opt/poolside/poolside-model-uploads`. The checkpoint is shipped as a tar archive. Extract it so that the checkpoint directory sits directly under the volume. For example: ```bash theme={null} tar -xf .tar -C /opt/poolside/poolside-model-uploads ``` The resulting directory name becomes the model's S3 path. In this example, the checkpoint is uploaded to `s3://poolside-models/`, which is the `s3_uri` value that you reference in [Step 4: Deploy Poolside model inference](#step-4-deploy-poolside-model-inference). `/opt/poolside/poolside-model-uploads` is the default location. If you customized the Poolside host volume location in `01-infra-rke2`, use the corresponding directory instead. 2. Run the following commands from the `03-poolside-model-upload` directory. **Air-gapped environment:** ```bash theme={null} TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform init TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform apply ``` **Internet-connected environment:** ```bash theme={null} terraform init terraform apply ``` 3. To upload additional or updated models later, repeat these steps. Uploads are additive and do not remove existing models from the deployment. The apply creates a Kubernetes job that syncs the volume into the `poolside-models` bucket and then waits for that job to complete, so the apply runs for as long as the upload itself. A checkpoint larger than 30 GB can take several minutes to upload from local disk. Terraform waits up to 20 minutes by default. To raise this limit for larger models or slower storage, set `overrides.upload_timeout` in `03-poolside-model-upload/terraform.tfvars`. If the apply times out with an error of the form `job: poolside-services/poolside-model-upload-... is not in complete state`, the upload job usually finishes shortly afterward. Wait for the job to report `Complete`, then re-run `terraform apply`. The re-run re-syncs incrementally and completes quickly. ```bash theme={null} export KUBECONFIG=/poolside-install/.kube/config kubectl get jobs -n poolside-services ``` ## Step 4: Deploy Poolside model inference The `04-poolside-inference` directory contains the Terraform module that deploys the inference containers used to serve Poolside models. 1. In the `04-poolside-inference` directory, update `terraform.tfvars` with the model details you want to deploy. The key for each entry in the `models` map is the model ID that clients use in API requests. Set `s3_uri` to the checkpoint directory uploaded in Step 3. Set `ingress_host_name` to the FQDN formed from a `poolside_ingress_hosts` prefix and `domain_name` in [Step 2: Install supporting infrastructure services](#step-2-install-supporting-infrastructure-services). If you use custom TLS certificates, the certificate SANs must include each model `ingress_host_name`. ```hcl title="Example: Model configuration" theme={null} deployment_name = "poolside-server" models = { agent = { s3_uri = "s3://poolside-models/" ingress_host_name = "poolside-models-agent.poolside.local" gpus = 2 replicas = 1 model_type = "agent" } } ``` 2. Run the following commands from the `04-poolside-inference` directory. **Air-gapped environment:** ```bash theme={null} TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform init TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform apply ``` **Internet-connected environment:** ```bash theme={null} terraform init terraform apply ``` ## Next steps: Post-installation configuration ### Configure local DNS Add hostname resolution on the deployment host. Replace the model ingress hostnames with the `ingress_host_name` values you configured in Step 4. If you expose multiple model ingress hostnames, include each hostname on the same line. ```bash theme={null} cat < seaweedfs.poolside.local seaweedfs-s3.poolside.local EOF ``` For example: ```bash theme={null} cat </poolside-install/.kube/config ``` ### Expected cluster state Model startup takes several minutes after the Step 4 apply completes. An init container pulls the checkpoint from S3-compatible storage, then vLLM loads the weights onto the GPUs. Watch for the model pod to show `READY 1/1` and `STATUS Running`: ```bash theme={null} kubectl get pods -n poolside-models -w ``` A healthy installation has all pods `Running` or `Completed` across the `kube-system`, `gpu-operator`, `poolside-cert-manager`, `poolside-registry`, `poolside-services`, and `poolside-models` namespaces: ```bash theme={null} kubectl get pods -A ``` Confirm that the model inference endpoint resolves to the deployment host: ```bash theme={null} getent hosts ``` Confirm that the model upload job completed successfully. The upload job runs in the `poolside-services` namespace: ```bash theme={null} kubectl get jobs -n poolside-services ``` You can confirm GPU usage with `nvidia-smi` on the host once the model pod is ready. ### Trust the deployment TLS certificate The installation host already trusts the deployment's certificates. [Step 2: Install supporting infrastructure services](#step-2-install-supporting-infrastructure-services) installs the self-signed CA into the host operating system's trust store, so `curl` and other clients running on the host verify TLS with no extra flags. On Ubuntu, the CA is under `/usr/local/share/ca-certificates/poolside/`. The location varies by operating system. External clients, such as developer workstations and CI systems, need the CA certificate to complete a trusted TLS handshake. Step 2 exports the CA to `/poolside-install/certs/poolside-ca.crt`. Copy that file to the external client and either install it into the client's trust store or pass it to `curl` with `--cacert`. If you provided your own externally valid certificates, use your own CA instead. ### Send a first request Poolside models are served over an OpenAI-compatible API at `https:///v1`. The model ID is the key of the `models` map from [Step 4: Deploy Poolside model inference](#step-4-deploy-poolside-model-inference), such as `agent` in the worked example. The endpoint URL is also available from `terraform output` in the `04-poolside-inference` directory. The commands below run on the installation host, which already trusts the deployment CA. From an external client, add `--cacert ` as described above. List the available models: ```bash theme={null} curl https://poolside-models-agent.poolside.local/v1/models ``` Send a first chat completion: ```bash theme={null} curl -X POST https://poolside-models-agent.poolside.local/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "agent", "messages": [{"role": "user", "content": "Write a one-line Bash command to count files in a directory."}], "max_tokens": 200 }' ``` A JSON response with a `chat.completion` object confirms that the deployment serves inference end to end. ## Troubleshooting The `kubectl` commands in this section use the kubeconfig exported in [Step 1: Install RKE2 on the host](#step-1-install-rke2-on-the-host). If you have not already done so in your current shell, export it first: ```bash theme={null} export KUBECONFIG=/poolside-install/.kube/config ``` ### Model pods stuck in `ContainerCreating` * Confirm that the host detects NVIDIA GPU devices: ```bash theme={null} lspci | grep -i nvidia ``` * Confirm that Kubernetes reports GPUs as allocatable: ```bash theme={null} kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}' ``` * Check model workload status: ```bash theme={null} kubectl get pods -n poolside-models ``` ### Models not loading * Confirm that the model checkpoint files were copied into `/opt/poolside/poolside-model-uploads`, or into the custom host volume location you configured. * Confirm that the model upload job completed successfully. * Check model initialization logs: ```bash theme={null} kubectl logs -c model-downloader -n poolside-models ``` ### Useful commands ```bash theme={null} # Check overall cluster status. kubectl get pods -A # Monitor model workloads. kubectl get pods -n poolside-models # Monitor supporting services. kubectl get pods -n poolside-services # Inspect pod and deployment details. kubectl describe pod -n kubectl describe deploy -n # View logs. kubectl logs -n # View recent events in a namespace. kubectl get events -n ``` ## Related resources * [On-premises installation prerequisites](/deployment/on-prem/prerequisites/overview) * [On-premises deployment](/deployment/on-prem/overview) * [Storage requirements](/deployment/on-prem/storage) * [Certified stacks](/deployment/on-prem/certified-stacks/overview) * [Upgrade on-premises](/deployment/on-prem/upgrade) * [Admin toolkit](/deployment/on-prem/admin) * [STIG hardening considerations](/deployment/on-prem/stig) * [On-premises deployment FAQ](/deployment/on-prem/faq) * [Supported configurations](/deployment/supported-configurations)