Skip to main content

​
Introduction

This page outlines the key performance indicators tracked by Poolside and explains the detailed events you can use to measure Poolside’s impact. Poolside tracks interactions between developers and the model. By integrating these metrics into your own data stores and business intelligence systems, you can understand how Poolside enhances your development processes.
Important: This page outlines the metrics available to you. Poolside does not have access to this data. For information about the metrics you share with Poolside, see Billing metrics.

​
Out-of-the-box metrics

Central to Poolside’s mission is enhancing your development velocity, productivity, and developer satisfaction. We understand that many organizations track metrics such as Cycle Time to measure the effectiveness of their development process. However, Cycle Time is a lagging indicator that often does not reflect developers’ day-to-day experiences. While these long-term metrics are important, they typically take months to reflect significant changes. At Poolside, we focus on leading indicators that demonstrate impact over shorter periods so you can observe improvements more quickly. These indicators fall into three core metric areas: impact, quality, and engagement.

​
Impact

Impact measures how Poolside improves the effectiveness of your development organization. External factors influence these metrics, so do not interpret them in isolation, but use them as key validation points to track. Developer throughput: We assess improvements in developer throughput by comparing productivity before and after Poolside adoption, tracking the number of daily pull requests (PRs) per developer. PRs are an excellent proxy metric for measuring developer productivity, as they represent the work an element of a team can accomplish daily. Developer satisfaction: We provide customizable qualitative surveys to measure how developers feel about Poolside. You can tailor these surveys to align with your organization’s needs and culture.

​
Quality

Poolside collects the following metrics as part of your deployment:
  1. Apply rate: The frequency of developers applying code completions without modifications.
  2. Review rate: The frequency with which developers use the review function in chat to understand proposed changes.
  3. Edit rate: The number of times developers modify code completion or chat suggestions.
  4. Discard rate: How frequently developers discard chat suggestions.
Poolside’s main objective is to make tasks simpler and more manageable with AI, allowing developers more time to focus on higher-value work. We aim to provide suggestions that developers can apply confidently by balancing effectiveness, relevance, and quality, increasing the number of times developers apply code without further modification.

​
Engagement

To understand how users are adopting different aspects of Poolside, we track the following engagement metrics:
  • Number of unique users engaging with the chat panel
  • Number of unique users utilizing code completion features
  • 90th percentile (P90) of prompts per user
  • Distribution of prompts across different coding languages
These metrics help you understand adoption rates across your user base and identify areas for potential improvement or additional support.

​
Metrics collection

Poolside uploads usage metrics in raw form to a customer-provided S3 bucket. A nightly aggregation cron job runs in Kubernetes to collect the previous day’s usage metrics and consolidate them into a single JSONL file: 
s3://<bucket>/telemetry/.../raw => s3://<bucket>/telemetry/aggregated/<date>.jsonl
The Metrics page in the Poolside Console displays data from these aggregated JSONL files, showing metrics from the previous day and earlier.

​
Event tracking

Poolside provides a comprehensive set of metrics out-of-the-box, but we understand that you may want to extend the scope of Poolside’s reporting. To facilitate this, Poolside exposes telemetry events so you can explore the data further and integrate it with existing reporting solutions.

​
Event structure

Each event captured by Poolside contains the following information: 
{
  "event_type": "string",
  "session_id": "string",
  "identity_id": "string",
  "tenant_sso_id": "string (optional)",
  "tenant_id": "string",
  "team_uuids": ["string"],
  "created_at": "timestamp",
  "assistant_version": "string (optional)",
  "assistant_host": "string (optional)",
  "assistant_host_version": "string (optional)",
  "metadata": {
    // Event-specific data including model information:
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    ...
  }
}

​
Event types

Poolside tracks various event types that correspond to user interactions and system responses. These events calculate the quality and engagement metrics described earlier and provide more detailed insight into how developers use the product. The events fall into two categories: chat events and completion events.

​
Chat events

prompt_started: Triggered when a user begins a new prompt to Poolside 
{
  "event_type": "prompt_started",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "parent_conversation_message_id": "string (optional)",
    "prompt_similarity": "float (optional)" // similarity to previous prompt
  }
}
prompt_completed: Triggered when Poolside finishes responding to a prompt 
{
  "event_type": "prompt_completed",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "response_time_ms": "number",
    "tokens_in": "number",
    "tokens_out": "number"
  }
}
response_regenerated: Triggered when the user requests a regeneration of a Poolside response. 
{
  "event_type": "response_regenerated",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "previous_message_id": "string"
  }
}
task_started: Triggered when a user initiates a new task. 
{
  "event_type": "task_started",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "task_id": "string",
    "task_type": "string",
    "task_description": "string"
  }
}
task_completed: Triggered when a task is marked as completed. 
{
  "event_type": "task_completed",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "task_id": "string",
    "duration_ms": "number",
    "result_status": "string"
  }
}
task_canceled: Triggered when a user cancels a task before completion. 
{
  "event_type": "task_canceled",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "task_id": "string",
    "time_spent_ms": "number",
    "reason": "string (optional)"
  }
}
task_reopened: Triggered when a previously completed or canceled task is reopened. 
{
  "event_type": "task_reopened",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "task_id": "string",
    "previous_status": "string",
    "reason": "string (optional)"
  }
}
rated_thumbs_up: Triggered when a user accepts and rewards a Poolside response. 
{
  "event_type": "rated_thumbs_up",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "response_id": "string"
  }
}
rated_thumbs_down: Triggered when a user declines and applies a penalty to a Poolside response. 
{
  "event_type": "rated_thumbs_down",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "conversation_message_id": "string",
    "conversation_id": "string",
    "response_id": "string",
    "feedback": "string (optional)"
  }
}

​
Completion events

completion_provided: Triggered when Poolside provides a code completion suggestion. 
{
  "event_type": "completion_provided",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "completion_id": "string",
    "file_path": "string",
    "language": "string",
    "completion_length": "number",
    "context_length": "number",
    "tokens_in": "number",
    "tokens_out": "number",
    "completion_time_ms": "number"
  }
}
completion_accepted: Triggered when a user accepts a Poolside code completion suggestion. 
{
  "event_type": "completion_accepted",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "completion_id": "string",
    "file_path": "string",
    "decision_time_ms": "number"
  }
}
completion_accepted_partially: Triggered when a user accepts only part of a Poolside code completion suggestion. 
{
  "event_type": "completion_accepted_partially",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "completion_id": "string",
    "file_path": "string", 
    "original_length": "number",
    "accepted_length": "number",
    "decision_time_ms": "number"
  }
}
completion_edited: Triggered when a user modifies a Poolside code completion suggestion before accepting. 
{
  "event_type": "completion_edited",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "completion_id": "string",
    "file_path": "string",
    "original_length": "number",
    "edited_length": "number",
    "edit_distance": "number",
    "edit_time_ms": "number"
  }
}
completion_rejected: Triggered when a user rejects a Poolside code completion. 
{
  "event_type": "completion_rejected",
  ...
  "metadata": {
    "model_id": "string",
    "model_name": "string",
    "model_type": "string",
    "completion_id": "string",
    "file_path": "string",
    "decision_time_ms": "number",
    "reason": "string (optional)"
  }
}
By leveraging these metrics and events, customers can gain deeper insights into their development processes, track the impact of Poolside on their workflows, and continuously optimize their use of the product. If you have questions about interpreting these metrics or need help with custom integrations, contact the Poolside team.