Kafka Fundamentals

Kafka is a distributed event streaming platform built to store, distribute, and replay ordered records at scale. People often meet it first as “a queue”, but that description is too small. Kafka is closer to a durable commit log shared by many producers and consumers.

1. Start with the record

The basic unit in Kafka is a record, often described as a message. A record has a key, value, timestamp, and optional headers. The key is important because it influences partition placement and therefore ordering guarantees for related events.

2. Understand topics and partitions

Records are written to topics. Each topic is split into partitions, and ordering is only guaranteed within a partition. This is one of Kafka’s most important design constraints. If two events must be processed in order, they need to land in the same partition, often by sharing a key.

3. Producers append, they do not update in place

Producers publish records to topics. They can batch writes for throughput and choose acknowledgements based on durability needs. Higher durability usually means more coordination with replicas and therefore more latency.

4. Consumers control their own progress

Consumers read records from partitions and track offsets that represent how far they have processed. This is a major difference from traditional messaging systems. Kafka keeps the log, and consumers decide what position they are at. That allows replay, reprocessing, and multiple independent consumer groups reading the same data.

5. Consumer groups provide scale and sharing

Within one consumer group, each partition is assigned to only one active consumer at a time. That gives parallelism while preserving per-partition order. Add more consumers than partitions and the extras sit idle. Partition count therefore becomes a scaling decision.

6. Replication provides durability and availability

Kafka brokers replicate partitions across multiple nodes. If the leader broker for a partition fails, another replica can take over. Durability depends on replication factor, in-sync replica policy, and producer acknowledgement settings.

7. Retention changes how you design systems

Kafka keeps records for a configured retention period or size limit, even after consumers read them. That makes it useful for audit logs, change data capture, stream processing, and event sourcing. It also means storage planning matters.

8. Use cases fit the log model

Kafka is strong for event-driven integration, CDC pipelines, analytics ingestion, asynchronous processing, and stream processing. It is weaker when you need per-message routing features more typical of traditional brokers or when operational simplicity matters more than throughput and replay.

The real mental model

Kafka is best understood as a durable shared log with partitioned ordering and consumer-managed progress. Once that clicks, its tradeoffs become easier to reason about. High throughput and replay power come from accepting partition-based ordering, explicit consumer state, and operational complexity at the cluster level.