RabbitMQ Video Processing Pipeline

The problem

Video processing needed to scale beyond synchronous API calls. Resizes had to be validated, queued, and processed reliably while staying compatible with existing asset workflows and team boundaries (API owners, workers, downstream movers).

What I designed

A Video Resize API for input validation, message serialization, and publishing jobs to a RabbitMQ quorum queue (a queue type designed for safety across nodes). That decoupled “accept the request” from “do the heavy work,” and let the pipeline absorb bursts without tying up web threads.

Coordination work

Not code in isolation. This was about aligning Assets Mover configuration, formats with the video team, worker consumer behavior, and demo planning for stakeholders. Backward compatibility mattered as much as the new architecture.

What I took away

Message-driven design shines when failure modes are explicit: poison messages, retries, consumers safe to run twice, and observability on queue depth. A practical lesson in re-architecting a pipeline without breaking consumers that still expected the old world.

That work fed into today’s Concourse to RabbitMQ Pipeline Migration.