Architecture

KalDB is a cloud-native search and analytics engine for high-volume logs, traces, and audit events. It uses Lucene for indexing and search, Kafka for durable ingest buffering, and S3-compatible object storage for durable indexed segments.

The architecture separates ingestion, indexing, query serving, coordination, and storage. Each tier can scale independently, and durable data is kept outside the compute nodes so components can be replaced or resized without moving all historical data with them.

How the architecture works

• Kafka stores durable, unindexed events until indexers consume them.
• S3 stores completed Lucene chunks after indexers finish building them.
• Indexers serve recent data while they are actively building chunks.
• Cache nodes serve completed chunks after loading them from S3.
• Query nodes are stateless and fan out each search to the indexers and cache nodes that may hold matching data.
• Cluster metadata tracks partitions, snapshots, replicas, cache assignments, and searchable endpoints.

The red arrows in the diagram represent the write path. The green arrows represent the read path. The dotted purple lines represent metadata reads and writes.

Write path

Applications send log, trace, or audit events through collectors such as Logstash, Vector, or the OpenTelemetry Collector. KalDB supports OpenTelemetry-based ingest, so teams can keep their existing OTel pipelines while sending telemetry into the KalDB write path. Collectors forward those events to preprocessor nodes, which validate and normalize the payloads, apply service-level controls, and partition the data into Kafka.

Indexer nodes consume assigned Kafka partitions and build Lucene indexes on local disk. Once an active chunk reaches its configured size or age limit, the indexer closes the chunk, uploads it to S3, and publishes metadata describing where that chunk lives and what time range it covers.

Read path

Clients query KalDB through the query service using OpenSearch-compatible APIs. The query service uses cluster metadata to find the partitions and snapshots that overlap the requested service and time range, then performs a scatter-gather query across active indexers and cache nodes.

Recent data is usually served by indexers because they still hold active or recently completed chunks locally. Older completed chunks are served by cache nodes, which download assigned Lucene indexes from S3 and expose them for search. The query service merges the shard responses and returns a single response to the client.

Components

App
The source application, service, or telemetry producer that emits logs, traces, or audit events.

Collector
An agent or forwarding layer that receives events from applications and sends them into the KalDB ingest path. In many deployments this is Logstash, Vector, the OpenTelemetry Collector, or a compatibility layer for existing logging pipelines. KalDB supports OpenTelemetry ingest for teams standardizing on OTel instrumentation and collectors.

Preprocessor
The ingest-facing service that validates timestamps and field types, applies rate limits, normalizes events into the format expected by indexers, and assigns records to Kafka partitions.

Kafka
The durable buffer for unindexed data. Kafka lets ingestion continue independently of indexing capacity and gives indexers a reliable source to replay from during recovery.

Indexer
The write-side search worker. Each indexer consumes one or more Kafka partitions, builds Lucene chunks, serves active and recent chunks for search, uploads completed chunks to S3, and publishes searchable metadata.

S3
The durable storage layer for indexed Lucene chunks. S3 keeps long-lived data outside the compute tier, allowing indexer and cache capacity to scale independently from retention size.

Query
The stateless search service. It receives client requests, finds the relevant shards through cluster metadata, fans the request out to indexers and cache nodes, merges results, and returns the response.

Cache
The read-side serving tier for completed chunks. Cache nodes download assigned Lucene chunks from S3 onto local disk and serve those chunks until they age out or are reassigned.

Cluster Manager
The coordinator for the cluster. It tracks nodes, partitions, snapshots, replicas, cache assignments, recovery work, and retention cleanup so data can move from indexers to S3 to cache nodes safely.

ZooKeeper
The metadata store used by KalDB components. ZooKeeper stores cluster state and supports coordination through Curator recipes used by the manager and serving components.