Rootly Private Agent

Connect AI SRE and the Rootly Agent in Slack to private services through an outbound-only, policy-controlled agent.

Early Preview: Rootly Private Agent is under active development and available only to approved customers. Features, configuration, limits, and APIs may change before general availability. Confirm the approved agent and backend versions with your Rootly representative before production use.

Rootly Private Agent gives Rootly AI SRE and the Rootly Agent in Slack controlled access to infrastructure that is not reachable from the public internet. The agent runs inside your network, opens an outbound gRPC connection over TLS and HTTP/2 to Rootly, and executes only registered, typed capabilities allowed by your local policy. Enrollment, credential refresh, provider registration, and work use the same secure endpoint.

Private Agent is one way Atlas, Rootly’s AI layer, reaches your private network. It extends AI connectors beyond public provider endpoints to clusters, telemetry backends, databases, and internal APIs you don’t expose publicly. AI SRE investigations and the Rootly Agent in Slack can use its capabilities, subject to their own access rules.

It is separate from Rootly Edge Connector. Edge Connector runs event-triggered HTTP or script actions. Private Agent serves bounded, typed diagnostics and explicitly allowlisted provider actions for AI SRE and the Rootly Agent in Slack; it does not provide arbitrary script execution.

How It Works

1. A user whose role includes Issue enrollment tokens selects Create enrollment token in AI SRE → Atlas → Private agents (AI & Agents → Private agents if your sidebar doesn’t have an AI SRE item). The token works once and expires after 24 hours.
2. The agent enrolls through gRPC over TLS, stores rotating credentials on its persistent volume, and registers its provider capabilities, health, and non-secret policy digest. It refreshes this snapshot every minute.
3. Rootly queues a typed invocation with an execution deadline.
4. The agent opens an authenticated bidirectional gRPC stream and advertises its available execution capacity, including per-instance slots for configured Prometheus, Loki, Tempo, Pyroscope, Argo CD, Kafka, Redis, Valkey, Elasticsearch, OpenSearch, PostgreSQL, MySQL, MCP, and HTTP providers. Rootly delivers eligible work over that connection and leaves excess work queued; the agent accepts only an exact provider, capability, and version it registered.
5. The provider enforces customer-local policy before calling the local service.
6. The agent sends a bounded structured result or typed error to Rootly through an authenticated gRPC call.

There is no inbound connection from Rootly to your network. Temporary disconnects leave eligible work queued until its deadline. Rootly dispatches new work only when an agent is recently online and its provider is healthy or degraded. Leased work uses heartbeats, bounded retries, and cooperative cancellation.

The connection carries cancellation even when all execution slots are occupied. After a temporary disconnect, the agent reconnects with backoff and reports still-running work rather than restarting it solely because the connection changed. There is no HTTP polling fallback. Coordinate upgrades from any earlier polling-based agent with your Rootly representative.

Terminal credential, local credential-storage, or malformed-protocol errors stop the work loop instead of reconnecting indefinitely. Check readiness and agent logs, correct the underlying configuration or storage problem, and coordinate credential recovery with an administrator. Do not delete persistent credential state as a routine reconnect step.

Each registered capability includes typed input and output JSON schemas, subject to the serialized schema limit. Rootly rejects an invalid provider snapshot with gRPC INVALID_ARGUMENT and keeps the agent’s last accepted snapshot active. Messages above the transport size limit are rejected with RESOURCE_EXHAUSTED.

Security Model

Access requires all of these independent checks:

1. Rootly authorizes the request for the actor making it.
2. The agent accepts only a registered typed capability.
3. Customer-managed local policy permits the operation.
4. The local service authorizes the agent credential, such as a Kubernetes ServiceAccount.

Rootly can narrow local access, but it cannot expand it. In the Kubernetes provider, Secrets, workload mutations, exec, attach, port-forward, token creation, impersonation, and RBAC administration are not supported capabilities. Optional authorization diagnostics use Kubernetes self-review APIs to evaluate the agent’s own access without persisting objects.

Rootly assigns every capability to one of three tiers:

• Safe read: Kubernetes API discovery, get, list, and watch, and the agent’s authorization self-reviews.
• Sensitive read: Kubernetes Pod logs, database diagnostics, internal HTTP GET, HEAD, and OPTIONS, internal MCP tools, and every Prometheus, Loki, Tempo, Pyroscope, Kafka, Elasticsearch, and OpenSearch capability.
• Write: internal HTTP POST, PUT, PATCH, and DELETE, and database execute_sql.

Which tiers a request can use depends on who makes it:

• Responder-started AI SRE investigations use the initiating user’s Private Agent permissions. Owners and admins can use all three tiers; users can use safe reads. Observers and no-access users cannot execute private capabilities. The agent’s local policy and provider credentials still apply.
• Automatic AI SRE investigations have no initiating user and run as the ai-sre system actor. When both AI SRE and Private Agent are enabled, they can use registered capabilities in every tier without applying an individual user’s role. Treat every capability your agent registers as potential unattended access and scope its provider credentials and allowlists accordingly.
• The Rootly Agent in Slack filters capabilities by the requesting user’s Rootly role before offering them and checks again at execution. Owners and admins can use all three tiers; users can use safe reads. Observers, no-access users, anonymous requests, and unrecognized-sensitivity contexts cannot execute private capabilities.

No Private Agent capability pauses for interactive approval, including those in the write tier.

Database execute_sql is a separate opt-in capability in the write tier because the agent does not classify statements. It remains available to unattended AI SRE investigations and authorized owner/admin user surfaces without an interactive approval pause, and its tool description requests read-only SQL. Always use a dedicated read-only database user because a write-capable identity could execute mutations.

Internal HTTP POST, PUT, PATCH, and DELETE methods are separate write capabilities and are registered only when the local configuration allowlists each method and path. They remain available to unattended AI SRE investigations and authorized owner/admin user surfaces without an interactive approval pause. Use a least-privilege upstream identity, expose only the required methods and path prefixes, and design enabled mutation endpoints for safe retry; see Internal HTTP mutation delivery.

The safe-read tier intentionally includes Kubernetes API discovery and the agent’s authorization self-reviews. An authorized AI SRE investigation, and any Slack user whose role allows safe reads, can inspect supported API resources and the selected provider identity’s effective RBAC. Rules reviews require a locally allowed namespace, but their raw output can reveal write verbs, unsupported resources, and cluster-scoped rules. This information does not grant additional executable capabilities. These tools evaluate the selected provider’s Kubernetes identity rather than the initiating user’s Kubernetes identity. Treat safe-read access, and read access to an investigated alert or incident, as access to this authorization information; see authorization diagnostics for scope and completeness limits.

The enrollment token is displayed once and stored by Rootly only as a digest. Agent access credentials are short-lived and refreshed with a rotating refresh credential. Revoking an agent invalidates both credentials immediately.

Rootly encrypts Private Agent invocation inputs, actor metadata, results, errors, and metrics at the application layer before storing them. Payload filtering applies to Private Agent request logs and transport tracing; it is not an exclusion from AI model traces or conversation history. Terminal transport records become eligible for scheduled deletion seven days after their last update, so job scheduling and backlog can delay removal.

Data returned to AI SRE can enter model context, evaluation traces, and investigation or conversation history. Deleting a transport record does not delete those downstream copies. See Private Agent data and retention for the shared explanation of transport filtering versus AI logging and retention.

The completed AI SRE report is shared with everyone who can read its alert or incident. Report evidence is not re-filtered against each later viewer’s Private Agent permission. Treat all readers of the target alert or incident as potential readers of Private Agent evidence that the investigation includes.

Deployment Modes

The supported early-access topology is a combined process with core and its configured providers in one Pod. Supporting builds can run multiple independently routed Kubernetes, Prometheus, Loki, Tempo, Pyroscope, Argo CD, Kafka, Redis, Valkey, Elasticsearch, OpenSearch, PostgreSQL, MySQL, internal MCP, and internal HTTP instances together. See Redis and Valkey setup for the cache diagnostics contract.

For the smallest Kubernetes credential boundary, deploy a separate in-cluster agent in each cluster with a distinct provider ID. A centralized combined agent can instead use mounted kubeconfigs for every cluster, or its own in-cluster ServiceAccount for one cluster and mounted kubeconfigs for additional clusters. Rootly routes each invocation to one exact agent and provider ID; it does not broadcast a query or choose a kubeconfig context. See Connect multiple clusters.

Confirm the approved early-preview agent and chart versions with your Rootly representative before deployment.

Install with Helm

In AI SRE → Atlas → Private agents (AI & Agents → Private agents if your sidebar doesn’t have an AI SRE item), select Create enrollment token and save the token to a local file. Rootly shows it only once. Then add the Rootly Helm repository and install the agent:

helm repo add rootly https://rootlyhq.github.io/helm-charts
helm repo update rootly

helm upgrade --install rootly-private-agent rootly/rootly-private-agent \
  --version 0.1.0 \
  --namespace rootly-private-agent \
  --create-namespace \
  --set-file enrollment.token=./enrollment-token

The enrollment token is used once. The chart stores refreshed agent credentials on a PersistentVolumeClaim, which should remain enabled for production deployments. For GitOps, create the enrollment Secret separately and configure enrollment.existingSecret and enrollment.secretKey instead of storing the token in Helm values.

Install on platforms without file-mounted secrets

For approved non-Kubernetes deployments on Unix-like container targets, the same Private Agent image includes a platform-neutral bootstrap mode. Use it when a scheduler can inject a secret-manager value as an environment variable but cannot mount that value as a file, such as Amazon ECS/Fargate, Nomad, or Docker Swarm. Kubernetes installations should continue to use the Helm chart and native Secret volume. Bootstrap is not supported by the Windows release binary.

Run the pinned agent image once as a short-lived init container:

/rootly-private-agent bootstrap \
  --config-file /bootstrap/config.yaml \
  --enrollment-token-file /bootstrap/enrollment-token \
  --owner 65532:65532

Configure the deployment with these boundaries:

1. Inject the base64-encoded, non-secret YAML configuration as ROOTLY_PRIVATE_AGENT_BOOTSTRAP_CONFIG_B64. In that YAML, set rootly.enrollment_token_file to /bootstrap/enrollment-token and set rootly.credential_state_file to the path on the separate durable volume.
2. Inject ROOTLY_PRIVATE_AGENT_BOOTSTRAP_ENROLLMENT_TOKEN directly from the platform secret store. Do not place the token in the task definition, deployment manifest, command line, or logs.
3. Mount an ephemeral volume at /bootstrap. Run bootstrap with a read-only root filesystem as UID 0, drop all Linux capabilities except CHOWN, and grant write access only to that volume.
4. Start the normal agent container only after bootstrap exits successfully. Use the exact same pinned image digest, run as UID/GID 65532, mount /bootstrap read-only, and set ROOTLY_PRIVATE_AGENT_CONFIG_FILE to /bootstrap/config.yaml. Do not pass either bootstrap environment variable to the long-running container.
5. Mount separate durable, encrypted storage at the configured rootly.credential_state_file. Its directory must be owned by or otherwise writable only to UID/GID 65532; for EFS, configure an access point with that POSIX identity and root-directory ownership. Preserve the storage across replacements so restarts use the rotating enrolled identity instead of enrolling again.

Bootstrap accepts at most 1 MiB of decoded YAML configuration and 4,096 bytes of enrollment-token input. It normalizes outer whitespace commonly added by secret files, rejects internal token whitespace, and writes the two files atomically with mode 0400. It does not contact Rootly or any configured provider. The one-time token is read by the normal agent only when durable credential state is absent.

Confirm the approved early-preview image version with your Rootly representative before using this mode. Platform-specific networking, durable storage, secret-store permissions, and workload identity remain the customer’s deployment responsibility.

Verify the container image

Each image is published for Linux AMD64 and ARM64 and signed by Rootly’s release workflow. The Helm chart pins the corresponding immutable image digest.

To verify the image and chart before deployment, install Cosign 3 or newer, resolve the digest, verify its OIDC-backed signature, and confirm that the chart renders the same digest:

VERSION=0.1.0-beta.5
CHART_VERSION=0.1.0
DIGEST="$(docker buildx imagetools inspect \
  "rootlyhub/rootly-private-agent:${VERSION}" \
  --format '{{.Manifest.Digest}}')"

cosign verify \
  --certificate-identity "https://github.com/rootlyhq/rootly-private-agent/.github/workflows/release.yml@refs/tags/v${VERSION}" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  "rootlyhub/rootly-private-agent@${DIGEST}"

helm template private-agent rootly/rootly-private-agent \
  --version "${CHART_VERSION}" \
  --set agent.controlPlaneEnabled=false \
  --show-only templates/deployment.yaml \
  | grep -F "rootlyhub/rootly-private-agent@${DIGEST}"

This verifies that the immutable multi-platform manifest was signed by Rootly’s release workflow for that version tag and that the selected chart version renders the same digest.

The agent exposes local health endpoints on port 8080 by default. If you override runtime.listen_address, configure your probes to use that address and port:

• GET /healthz for liveness
• GET /readyz for control-plane and provider readiness
• GET /version for build and component information

These endpoints are intended for local probes. They do not need an externally reachable Service or Ingress.

Manage Agents

Paths on this page use the consolidated AI SRE navigation, which is rolling out. If your sidebar still shows AI & Agents, open AI & Agents → Private agents. See Where AI Settings Live.

Open AI SRE → Atlas → Private agents to:

• Create a one-time enrollment token
• Review online status, last-seen time, version, and registered providers
• Review each provider’s type, health, last check, and advertised capabilities
• Review the number of capabilities advertised by an agent
• Edit an agent’s name and description
• Revoke an agent immediately

The Private agents card appears only when Private Agent and AI SRE are both enabled for the selected team. Your Rootly account team enables them; see Getting Started for AI SRE availability.

Each action needs its permission in the Private Agents group of your Incident Response role:

Permission	Allows
View agent inventory	Opening the page and reviewing agents and providers
Issue enrollment tokens	Creating enrollment tokens
Edit agent metadata	Changing an agent’s name and description
Revoke agent credentials	Revoking an agent

The default owner and admin roles grant all four. The default user and observer roles grant View agent inventory only. The browser inventory needs View agent inventory. Opening it through AI & Agents requires an Incident Response seat; without one, a user with that permission can append /account/private-connect/agents to the Rootly app URL. The other three permissions also work on their own through the API. For the rest of the AI SRE permission model, see AI & Agents and AI SRE permissions.

An agent’s description helps Rootly choose between agents when more than one is online, so describe what the agent reaches. Don’t include secrets, personal data, or other sensitive information.

An agent is shown as online when it is active and has contacted Rootly within the last two minutes.

Agent connectivity and provider health are separate: an online agent can have a healthy Kubernetes provider and an unhealthy external-provider instance. Healthy means the provider reports availability; Degraded means it reports limited availability but can still receive authorized work. Unhealthy providers are excluded from new dispatch. Overall /readyz returns not_ready if any provider is not healthy, including a degraded provider; readiness is distinct from dispatch eligibility.

The provider-health UI marks observations older than two minutes as Stale and does not present offline or revoked agents as healthy. Missing, invalid, or implausibly future-dated observations appear as Unavailable. These observation-freshness labels are display-only: dispatch separately checks recent agent connectivity, reported provider availability, authorization, and local policy. A stale display alone does not disable an otherwise eligible provider. Use Refresh to retrieve the latest stored snapshot; this does not force a new upstream health probe. Displayed capabilities are agent-reported, not a guarantee that the current user or local policy permits their execution.

Automate Management Through the Public API

These endpoints require a backend build with the Private Agent management API enabled. Confirm availability with your Rootly representative. They do not replace the agent’s gRPC protocol.

Deployment automation can use the existing Rootly API service at https://api.rootly.com with a Rootly API key whose user or service account has the matching Private Agents role permission. Private Agent and AI SRE must both be enabled for the account. OAuth scopes don’t grant access to these endpoints.

Endpoint	Purpose
POST /v1/private_agents/enrollment_tokens	Create a one-time token for agent enrollment; does not register an agent.
GET /v1/private_agents	List agent identity and connectivity, including offline and revoked agents.
GET /v1/private_agents/{id}	Retrieve one agent with its last-reported provider health, policy, and capabilities.
PATCH /v1/private_agents/{id}	Update an agent’s name and description. Other attributes are ignored.
POST /v1/private_agents/{id}/revoke	Invalidate agent credentials and remove its provider routing registrations.

Send Authorization: Bearer <ROOTLY_API_KEY>, Accept: application/vnd.api+json, and Content-Type: application/vnd.api+json over HTTPS. See API authentication and conventions. Neither POST requires a request body. The enrollment-token response contains data.attributes.token and data.attributes.expires_at, uses Cache-Control: no-store, and returns the secret only once. The token expires after 24 hours and can be consumed once. Repeating token creation produces a different token; this operation is not idempotent.

The two enrollment steps use different credentials:

1. Your trusted deployment automation calls the public management API with a Rootly API key to obtain an enrollment token.
2. It securely supplies only that one-time token to the agent. The agent exchanges it through gRPC Enroll for rotating agent credentials and creates its registered identity.

Do not place your Rootly management API key in the agent configuration, print the enrollment response in CI logs, or commit the token to source control. Agent access and refresh credentials cannot authenticate to the management API. Creating enrollment tokens through the API is optional; the UI remains available for manual setup.

List requests accept page[number] and page[size] (default 50, maximum 1000) and return pagination metadata. Lists omit provider snapshots and do not load encrypted credentials or snapshots from the database; retrieve individual agents when provider details are needed. Provider observations can be stale, and online does not imply that every provider is healthy. Credentials and capability input/output schemas are never returned by inventory endpoints.

Revocation is safe to repeat and retains the agent record and invocation history. It prevents new authenticated work, but an operation already executing inside the customer network may not stop immediately. A revoked installation needs a new enrollment token to enroll again. Management access is account-scoped; inaccessible agent IDs return 404 without revealing whether they exist.

Next Step

See Private Agent for Kubernetes for local policy, RBAC, compatibility, Pod logs, and sizing guidance.

See Private Agent for Prometheus for multi-instance configuration, query tools, authorization, and performance limits.

See Private Agent for Loki for bounded log queries, label discovery, scan-cost controls, patterns, tenancy, and TLS.

See Private Agent for Grafana Tempo for bounded TraceQL searches, trace retrieval, attribute discovery, trace metrics, tenancy, and TLS.

See Private Agent for Grafana Pyroscope for bounded profile discovery, timelines, flame graphs, comparisons, label scope, tenancy, and TLS.

Argo CD support is build-specific; confirm availability with your Rootly representative before deploying it. See Private Agent for Argo CD for project scope, token RBAC, resource redaction, and compatibility.

See Private Agent for Kafka for multi-cluster routing, topic scope, Apache Kafka, Amazon MSK IAM, Redpanda, and bounded diagnostics.

See Private Agent for Redis and Valkey for bounded cache health, memory, replication, slowlog, and latency diagnostics.

See Private Agent for Elasticsearch and OpenSearch for index allowlists, bounded Query DSL, authentication, AWS SigV4, and compatibility.

See Private Agent for PostgreSQL and MySQL for read-only users, schema-scoped diagnostics, and bounded SQL execution.

See Private Agent for Internal MCP Servers for exact tool allowlists, local credentials, protocol restrictions, and interoperability testing.

See Private Agent for Internal HTTP APIs for fixed-origin routing, method and path allowlists, credentials, TLS, and response limits.

See Private Agent Limits for the canonical runtime, adapter, schema, and AI-context defaults and ceilings.

Roadmap Scope

Split deployments and providers other than Kubernetes, Prometheus, Loki, Tempo, Pyroscope, Argo CD, Kafka, Redis, Valkey, Elasticsearch, OpenSearch, PostgreSQL, MySQL, internal MCP, and internal HTTP are outside the release described here. Do not plan a deployment around an unannounced provider or topology; confirm supported builds and capabilities with your Rootly representative.

Tempo is build-specific and coordinated through Rootly support.

Related Pages

Atlas Overview

Rootly’s AI layer, its entry points, and the context available to each.

AI Connectors

Connect the providers AI SRE and the Rootly Agent reach over their public endpoints.

Evidence Sources

Everything an investigation reads, and the access boundaries around it.

Running an Investigation

Every way an investigation starts, including automatic runs with no responder in the loop.

Frequently Asked Questions

Which Rootly AI features use private agents?

Rootly AI SRE investigations and the Rootly Agent in Slack. Each applies its own access rules, as Security Model describes.

Do investigations use my role when I start one?

Yes, for an investigation you start. Your Private Agent permissions determine which capabilities AI SRE can call. Automatic investigations instead use the ai-sre system actor and can call registered capabilities allowed by the agent’s local policy without inheriting a responder’s role. Scope provider credentials and local policy for unattended use.