Concourse to RabbitMQ Pipeline Migration

The shift

We’re moving a Go-based pipeline off Concourse (a Continuous Integration (CI) orchestrator) and onto RabbitMQ (a message broker). Instead of CI jobs driving every step, work goes into queues. Background workers pick up jobs, finish them, and retry with clear rules when something fails.

Why RabbitMQ

Concourse served us well for orchestration. This workload benefits from durable queues, scaling consumers up and down, and messaging patterns built for handoffs. RabbitMQ separates “enqueue work” from “process work,” which matches how the Go services already think about the problem.

Dead-letter exchanges

A core part of the design is dead-letter exchange (DLX) routing. Messages that fail processing or exceed retry limits land in a separate queue operators can inspect instead of getting buried in logs. Poison messages get quarantined. Transient failures retry with bounds. Operators can see what is stuck and why.

ADR approved

The architecture went through formal Architecture Decision Record (ADR) review and approved. I write about that process in Demystifying ADRs. The ADR itself locks in queue topology, failure handling, compatibility constraints, and rollback thinking before implementation spreads.

What I’m focused on now

Incremental rollout: producers publishing safely, consumers safe to run twice (idempotent), observability on queue depth and dead-letter volume, and a migration path that does not require one maintenance window for everything.

Tie-in to earlier work

This connects to prior RabbitMQ Video Processing Pipeline work (validation, serialization, quorum queues, and cross-team coordination), applied now at the platform pipeline layer in Go.