Tenant isolation without separate clusters

KalDB runs many tenants in one cluster without letting them interfere with each other. Each service gets its own Kafka partitions, its own indexers, and its own ingest quota, and searches only reach the partitions that hold that service’s data.

How KalDB isolates tenants inside one clusterTwo services, payments and web, are separate datasets. The preprocessor routes each service's logs to its own Kafka partition, and each partition has its own indexer. When web spikes, only web's partition and indexer fall behind. A payments search from OpenSearch Dashboards only reaches the payments indexer.Datasets (in ZooKeeper)payments→ partition 14 MB/sweb→ partition 28 MB/sLogstashpaymentsLogstashwebtraffic spikePreprocessorroute by serviceKafka partition 1Kafka partition 2Indexer 1payments onlyIndexer 2web onlycatching upQueryOpenSearchDashboardsOnly partition 2 backs up. Payments is untouched.Traffic above web's quota is dropped at the preprocessor.GET payments/_searchSearches 1 of 2 partitions. Indexer 2 is never asked.
Tenants payments: fresh · web: fresh payments: fresh · web: falling behind The payments search only touched payments data.
  1. 1. Each service is a dataset. A dataset maps a service name to its own Kafka partitions and sets its ingest quota. KalDB keeps this metadata in ZooKeeper, and the preprocessor reads it.
  2. 2. The preprocessor routes by service. Logstash sends logs for payments and web. The preprocessor matches each batch to its dataset and writes it only to that dataset's partitions.
  3. 3. One indexer per partition. Indexer 1 only ever sees payments logs and Indexer 2 only ever sees web logs. The two tenants never share indexing capacity.
  4. 4. Web spikes. The extra web traffic lands on partition 2, and only Indexer 2 falls behind. Payments logs stay fresh. Anything above web's quota is dropped at the preprocessor.
  5. 5. Searches stay in their lane. A user searches payments from OpenSearch Dashboards. The query node looks up which partitions payments used in that time range and only asks Indexer 1.

The problem

Shared logging clusters have a noisy-neighbor problem. One service ships a bad deploy, its log volume jumps, and every other team’s logs fall behind with it. Heavy searches from one team slow down searches for everyone else.

The usual fix is a separate cluster per large tenant. That works, but at scale it means running and upgrading dozens or hundreds of clusters, each sized for its own peak, with the tooling and on-call load to match.

How it works

  1. Each service is a dataset. A dataset has a name, an owner, the service name it matches, an ingest quota (throughputBytes, in bytes per second), and the Kafka partitions it writes to. Datasets are stored in ZooKeeper and managed through the Manager API.
  2. The preprocessor routes by service. Clients send logs to the OpenSearch-compatible _bulk API, using the service name as the index. The preprocessor matches each batch to its dataset and writes it to one of that dataset’s partitions. Logs for a service that has no dataset are dropped instead of landing in someone else’s partition.
  3. One indexer per partition. Every indexer consumes exactly one Kafka partition. If a tenant has dedicated partitions, its indexers only ever see its logs.
  4. Quotas are enforced at the front door. The preprocessor rate-limits each dataset to its throughputBytes. Traffic above the quota is dropped and counted, so one tenant’s spike can’t flood the cluster.
  5. Searches only reach the tenant’s partitions. A search for an index (dataset) goes to the query node, which looks up the partitions that dataset used during the requested time range and only searches data from those partitions. Inside each chunk, results are also filtered by service name, so a search never returns another tenant’s logs.

Dedicated and shared partitions

Each dataset either has dedicated partitions, used by no other dataset, or shared partitions. Shared is the default and suits small services that don’t need an indexer of their own. Give large or critical tenants dedicated partitions for full isolation of indexing capacity. Searches stay scoped either way: on shared partitions, KalDB filters by service name inside each chunk.

Why it matters

  • No noisy neighbors. A spike or bad deploy in one service only affects that service’s partitions and indexers.
  • One cluster instead of many. Tenants get workload isolation without a separate cluster for each, so there is one system to upgrade, monitor, and size.
  • Faster, cheaper searches. Searches only touch the partitions that can contain matching data, instead of fanning out across the whole cluster.
  • Predictable ingest. Per-tenant quotas cap how much any one service can push, and every dropped message is counted with the reason.

Trade-offs

  • Shared partitions share capacity. Tenants on the same shared partitions still compete for the same indexers. Move a tenant to dedicated partitions when it needs strict isolation.
  • Dedicated partitions cost an indexer each. Each dedicated partition needs its own indexer, even if the tenant is small.
  • Quotas drop data. Logs above a tenant’s quota are dropped, not buffered. Size throughputBytes for the tenant’s expected peak, and alert on dropped messages.
  • Services need a dataset. Logs for an unknown service name are dropped, so new services must be provisioned before they send logs.

Configuration

Create a dataset, then assign it a quota and partitions with the Manager API:

POST /slack.proto.astra.ManagerApiService/CreateDatasetMetadata
{
  "name": "payments",
  "owner": "payments-team",
  "serviceNamePattern": "payments"
}
POST /slack.proto.astra.ManagerApiService/UpdatePartitionAssignment
{
  "name": "payments",
  "throughputBytes": "4000000",
  "partitionIds": [],
  "requireDedicatedPartition": true
}

Leave partitionIds empty to let the manager pick partitions automatically based on the dataset’s throughput, or list specific partition IDs. Set requireDedicatedPartition to true to give the dataset partitions of its own.

To watch quota enforcement, track these preprocessor metrics, which are tagged by service and drop reason:

  • preprocessor_rate_limit_messages_dropped_total
  • preprocessor_rate_limit_bytes_dropped_total

Related