00. Celery — ELI5

Celery is a kitchen with a ticket rail. Waiters drop tickets on the rail; cooks pull them, work them, and mark them done. Tickets that fail get retried; tickets that age get escalated. The dining room never waits for the kitchen.

---

Picture a restaurant. The waiter takes the order and hands it to the kitchen as a paper ticket pinned to a rail. The waiter walks back to serve another table. Cooks watch the rail, pull tickets, cook the food, plate it, ring a bell. A runner takes the plate to the table.

Celery is exactly this pattern, applied to a web app. The web handler (waiter) needs to do something slow — send an email, generate a PDF, run a daily report, call a slow third-party API. Instead of holding up the request, the handler puts a task ticket on a queue (the rail). A separate process pool (cooks) consumes the queue and executes the task. The web request returns to the user immediately; the work happens in the background.

The ticket rail in Celery is the broker — usually Redis or RabbitMQ. The cooks are workers — Python processes that consume the queue. The result, if anyone wants it, can be stored in the result backend — Redis, a database, or nowhere if you don't care.

That's it. Producers put tasks on the broker. Workers consume and execute. The web app and the heavy lifting are decoupled.

---

The recurring vocabulary

Name	What it is
broker	the message queue Celery uses to pass tasks (Redis, RabbitMQ, SQS)
worker	the Python process that pulls tasks and executes them
task	a Python function decorated with @app.task
queue	a named channel within the broker for routing tasks
result backend	optional store for task return values (Redis, database, RPC)
acks_late	acknowledge the task only after it completes (vs. on receipt)
prefetch	how many tasks a worker pulls in advance
chain / group / chord	task composition patterns
beat	the scheduler process that triggers periodic tasks
flower	the monitoring UI for Celery

---

The picture

       ┌────────────────────────────┐
       │   web app (Django, etc.)   │
       │  send_welcome_email.delay()│
       └──────────────┬─────────────┘
                      │ enqueue
                      ▼
            ┌──────────────────┐
            │   broker         │   Redis / RabbitMQ
            │  (queues: emails │
            │   reports, ...)  │
            └────────┬─────────┘
                     │ consume
        ┌────────────┼────────────┐
        ▼            ▼            ▼
   ┌────────┐   ┌────────┐   ┌────────┐
   │ wkr 1  │   │ wkr 2  │   │ wkr 3  │     pool of workers
   └────┬───┘   └────┬───┘   └────┬───┘
        │            │            │
        └──── execute task ───────┘
                     │
                     ▼ (optional)
            ┌──────────────────┐
            │  result backend  │
            └──────────────────┘

The web app produces; the broker queues; the workers consume. The result backend is optional and often skipped.

---

Two facts that surprise new Celery users

The web app and Celery share code, not memory. Workers boot the same Django app the web does. A task imports models, uses the ORM, hits the database. But the worker is a separate process — restarting the web does not restart workers, and a code change requires restarting both.

Tasks are not guaranteed-once. Default Celery is "at-least-once": a task may run more than once on failure. If your task sends an email, it can send twice. Tasks must be idempotent — designed so running twice produces the same outcome as running once.

---

What this module covers

1. 01-broker-worker-protocol-internals.md — How tasks become messages, how workers consume, the protocol between broker and worker, what acks_late and prefetch really do.
2. 02-tasks-queues-day-to-day.md — Defining tasks, routing to queues, composition (chain, group, chord), periodic tasks via beat.
3. 03-retries-monitoring-prod-gotchas.md — Retries, idempotency, monitoring, the failure modes (lost tasks, duplicate tasks, queue depth), the gotchas that surface only in production.

---

What this module is not about

• Choosing between Celery and alternatives (RQ, Dramatiq, Arq, Huey) — covered briefly in chapter 3.
• Building a custom worker pool from scratch — out of scope.
• AWS SQS specifics as a broker — covered in 07_sqs.

---

Bridge. The kitchen-and-tickets picture is enough to believe Celery works. To debug a task that runs three times when it should run once, you need to see the protocol. Chapter 01 opens the broker-worker handshake. → 01-broker-worker-protocol-internals.md