Learning Kafka

If RabbitMQ is a kitchen ticket rail (take an order, finish it, move on), Kafka is more like a security camera recording that never stops.

Events get appended to a log. Anyone can come back later and watch from wherever they left off. Nothing disappears just because someone already read it.

The simple idea

Queue mindset: deliver a task, mark it done, gone.

Log mindset: record what happened, keep the tape, let many readers catch up at their own speed.

flowchart TB
  subgraph queue["Queue style: task is done and gone"]
    Q1[Job] --> W[Worker finishes]
    W --> X[Job removed]
  end

  subgraph log["Log style: history stays"]
    E1[Event] --> L[Append to log]
    L --> R1[Reader A at their pace]
    L --> R2[Reader B at their pace]
    L --> R3[Reader C rewinds and replays]
  end

That replay part is the superpower. Need to rebuild a dashboard from last Tuesday? Re-read the log. New service wants the same stream of facts? Attach another reader. You do not re-send everything by hand.

When Kafka fits

Good fits:

• Audit trails: “show me everything that happened to this account”
• Analytics pipelines: many teams reading the same firehose of events
• Systems that need rewind: new app version replays history to warm up

Less ideal when you just want one worker to pick up the next job and ack it. That is queue territory. See Learning RabbitMQ for that side of the house.

Neither is universally better. They answer different questions.

flowchart LR
  P[Producer writes events] --> K[Kafka log]
  K --> C1[Consumer group A]
  K --> C2[Consumer group B]
  K --> C3[New service replays from start]

Consumer group: fancy term for “a team of readers sharing the load.” Each group tracks its own progress through the tape.

How I am learning

This one is background depth for now: reading, local experiments, comparing feel to queue-based patterns I am already learning.

At work I got my feet wet implementing Kafka Cruise Control for our cluster: automated load balancing when brokers get uneven, instead of someone hand-moving partitions at 2 AM. That was ops depth, not application design, but it forced me to understand how the cluster actually behaves under load.

When a problem needs durable history or multiple independent readers, that is my cue to go deeper.

Hands-on lab: Event Patterns, with RabbitMQ and Kafka running side by side so I can see the difference instead of reading about it.

Where this goes

I use Kafka at work. This track is about closing the gaps of knowledge between me and our systems.