01. ORM, request cycle, querysets — what Django actually does per request¶
~16 min read. One innocent Order.objects.filter(status='paid')[:25] looks like one SQL query. It is not always — and the "is not always" is most Django performance bugs. This chapter opens the request lifecycle, the ORM, lazy QuerySets, and N+1 — the four things every Django interview eventually circles back to.
Builds on: 00-eli5.md.
The hallway-and-doors picture is enough to believe Django works. To debug a slow endpoint or explain a senior question, you need to see what each door actually does. We will follow one request — GET /api/orders/?status=paid&limit=25 returning JSON — all the way through, and watch the ORM build its SQL, hit the database, and serialise the response.
1) The request cycle, in seconds¶
A new request arrives at gunicorn. Walk through what happens in order.
TIME LAYER ACTION
──── ────────────────────────── ────────────────────────────────────────
0.0ms gunicorn worker accepts socket, parses HTTP
0.1ms WSGI handler constructs Django HttpRequest object
0.2ms middleware (entering) SecurityMiddleware, SessionMiddleware,
AuthenticationMiddleware, CSRF, ...
0.8ms URLconf resolver matches /api/orders/ to OrdersView.as_view()
1.0ms view dispatch calls view.get(request)
1.1ms view body Order.objects.filter(...).select_related(...)
— builds QuerySet, no SQL yet
1.2ms serializer / template iterates QuerySet — SQL fires now
3.5ms database round trip Postgres returns 25 rows
3.7ms view returns HttpResponse JSON-encoded body
3.8ms middleware (exiting) response gets cookies, headers, gzip
4.0ms gunicorn writes response to socket
The view body and the database round-trip dominate. Everything else is single-digit microseconds. When you optimise Django, you almost always optimise either the queries (chapter 2) or the work the view does between query result and response (this chapter).
2) The ORM is lazy — what QuerySet really is¶
A QuerySet is not a list of rows. It is a plan for a query that has not run yet. The plan is mutable through method chaining.
qs = Order.objects.filter(status='paid')
# No SQL has run. qs is a QuerySet object holding {model: Order, where: status='paid'}.
qs = qs.filter(created_at__gte='2026-01-01')
# Still no SQL. The plan has another WHERE clause added.
qs = qs.select_related('customer')
# Still no SQL. The plan now says "join the customer table".
qs = qs.order_by('-created_at')[:25]
# Still no SQL. The plan now has ORDER BY and LIMIT.
for order in qs: # <-- HERE the query runs.
print(order.customer.email) # because select_related joined, no extra query
A QuerySet evaluates — runs the SQL — when you do any of these:
- iterate over it (
for x in qs) - call
len(qs)orbool(qs) - call
list(qs) - slice with a step (
qs[0:10]is still lazy;qs[::2]forces evaluation) - call
.first(),.last(),.exists(),.count(),.aggregate() - pickle it
- evaluate it in a Django template
The same QuerySet evaluates only once per Python execution — results are cached on the QuerySet object. But a new QuerySet (from .filter() or any chain) is a separate cache.
qs = Order.objects.filter(status='paid')
list(qs) # SQL runs once. Results cached on qs.
list(qs) # No SQL. Reads from qs._result_cache.
list(qs.filter(...)) # New QuerySet. SQL runs again.
This laziness is what makes Django ergonomic. It is also what causes the next thing.
3) N+1 — the most common Django performance bug¶
Take a view that lists 25 orders and their customer emails.
def order_list(request):
orders = Order.objects.filter(status='paid')[:25]
return JsonResponse([
{'id': o.id, 'customer': o.customer.email}
for o in orders
])
Looks innocent. Run it. Watch the database.
-- query 1: the orders
SELECT * FROM orders WHERE status='paid' LIMIT 25;
-- query 2 through 26: one per row
SELECT * FROM customers WHERE id = 421;
SELECT * FROM customers WHERE id = 422;
SELECT * FROM customers WHERE id = 421; -- duplicate; ORM doesn't dedupe
SELECT * FROM customers WHERE id = 504;
...
Twenty-six queries for one request. The pattern is:
- 1 query to fetch the 25 orders.
- N queries, one per order, to fetch its customer (because
o.customeris a foreign key access, and Django's ORM lazily fetches it the first time you touch it).
That is the N+1 query problem. It is the single most common Django performance bug. At 25 rows it adds 20ms. At 250 rows and 50ms-per-query (small cloud Postgres), the request goes from 60ms to 12 seconds. The page hangs. The pager fires.
Fix:
def order_list(request):
orders = Order.objects.filter(status='paid').select_related('customer')[:25]
return JsonResponse([
{'id': o.id, 'customer': o.customer.email}
for o in orders
])
select_related('customer') tells the ORM to JOIN the customers table in the same query. One SQL statement returns all 25 orders and their customer rows. The query becomes:
SELECT orders.*, customers.*
FROM orders LEFT JOIN customers ON orders.customer_id = customers.id
WHERE orders.status = 'paid' LIMIT 25;
26 queries become 1. The request returns in 50ms instead of 12 seconds.
Teacher voice. Every Django code review should ask, for every foreign-key access in a list view, "is the related object prefetched?" The cost of forgetting is borne in production by users, not the author.
4) select_related vs. prefetch_related¶
select_related works for foreign keys and one-to-one. It uses SQL JOINs. One query, denormalised result.
prefetch_related works for reverse foreign keys and many-to-many. It cannot use JOINs (those would multiply rows). Instead, it runs two queries: the main one, and a second to fetch all related rows by IDs.
# A blog with posts and their tags (many-to-many).
posts = Post.objects.all().prefetch_related('tags')
Two queries regardless of how many posts. Without prefetch_related, every post.tags.all() access fires another query — N+1 again, just on a many-to-many.
The choice:
- One foreign key, single related row per parent →
select_related. - Many related rows per parent (reverse FK, M2M) →
prefetch_related.
Mistakes:
- Using
select_relatedon a M2M relationship — Django will raise an error. - Using
prefetch_relatedfor a single FK — works, but does two queries when one would suffice.
5) When the ORM bites — the queries that aren't what you think¶
The ORM is convenient, but its emitted SQL is not always what a senior engineer would have written.
Q objects vs. chained .filter(). Two .filter() calls become AND clauses. To express OR, use Q:
# AND — paid orders created this year
Order.objects.filter(status='paid').filter(created_at__year=2026)
# OR — paid OR shipped
from django.db.models import Q
Order.objects.filter(Q(status='paid') | Q(status='shipped'))
exclude() is not NOT filter(). Subtle: exclude(a=1, b=2) is "exclude rows where a=1 AND b=2", not "exclude rows where a=1 OR b=2". The de Morgan flip catches people.
.distinct() on a JOIN. When you .filter() across a M2M, you can get duplicate rows. Add .distinct(). The cost is a SELECT DISTINCT in the SQL.
.count() is a full COUNT(*). For pagination, the ORM's default paginator runs COUNT(*) first. On a large table with a filter, this can be slower than the actual page fetch. For analytics-heavy lists, use CursorPagination or skip the count.
.update() does not call save(). Order.objects.filter(...).update(status='paid') runs one UPDATE in SQL and skips Django signals (pre_save, post_save). If you have a post_save signal that emails the customer, it does not fire. update() is bulk; save() is per-row.
Raw SQL through .raw() or .extra(). When the ORM cannot express what you need (window functions, recursive CTEs, complex JOINs), drop to raw SQL. It bypasses much of the ORM safety; you take responsibility for parameter binding and SQL injection.
6) The query plan — .query and EXPLAIN¶
The ORM's compiled SQL is accessible.
>>> qs = Order.objects.filter(status='paid').select_related('customer')[:25]
>>> print(qs.query)
SELECT "orders"."id", "orders"."status", "orders"."customer_id", "customers"."id", "customers"."email"
FROM "orders" LEFT OUTER JOIN "customers" ON ("orders"."customer_id" = "customers"."id")
WHERE "orders"."status" = 'paid'
LIMIT 25
Run that through Postgres's EXPLAIN ANALYZE to see the actual plan, scan choice, index use, and row counts. Django's connection.queries (in DEBUG mode) shows every query the current request fired, with timings. Use it during development to find N+1.
In production, use django-debug-toolbar (development only — it slows requests) or instrumentation like django-silk, OpenTelemetry, or Sentry's Django integration to capture per-request query counts and timings.
7) Middleware: the pre/post hooks¶
Middleware is a list of classes (or functions) that wrap every request. Each class gets the request on the way in and the response on the way out.
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'myapp.middleware.RequestIdMiddleware',
]
Order matters. Authentication middleware reads the session, so it must run after SessionMiddleware. CSRF reads the cookie set by Session, so it runs after both. The list is read top-to-bottom on the way in and bottom-to-top on the way out.
Custom middleware is a class with __init__(self, get_response) and __call__(self, request):
class RequestIdMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
request.id = uuid.uuid4().hex
response = self.get_response(request)
response['X-Request-Id'] = request.id
return response
Anything you want to run for every request — request ID, logging, throttling, feature-flag injection, multi-tenant scoping — lives here.
8) Settings, apps, and the import-time graph¶
Django boots by importing settings.py, then walking INSTALLED_APPS and importing each app's apps.py, then resolving the URLconf. The import order is fixed; circular imports between apps will surface here, not at runtime.
A common gotcha: importing models at the top of a non-models file before Django is fully loaded. The fix is lazy imports — import inside functions, or use apps.get_model('app_label', 'ModelName').
def some_signal_handler(sender, **kwargs):
from myapp.models import Order # lazy — Django is fully loaded by now
...
9) Signals — pub/sub on model events¶
Django fires signals on model lifecycle events: pre_save, post_save, pre_delete, post_delete, m2m_changed. Other places fire them too (request_started, request_finished).
from django.db.models.signals import post_save
from django.dispatch import receiver
@receiver(post_save, sender=Order)
def order_saved(sender, instance, created, **kwargs):
if created:
send_welcome_email.delay(instance.customer_id)
Signals are convenient and dangerous. Convenient because they decouple "an Order was saved" from "send an email." Dangerous because:
- They are synchronous by default. The email send blocks the save's transaction.
- They run inside the database transaction. If the email send fails, the transaction may roll back — or not, depending on your error handling.
- They are easy to forget. A new developer adds an Order from a management command and wonders why the email did not fire (they used
Order.objects.create()— yes, signals fire — but if they usedbulk_create()they would not).
Most senior teams move side effects out of signals into explicit service functions or Celery tasks. Signals are kept for cross-app integration (e.g., django-allauth listens for user creation), not for primary business logic.
10) The threaded example — finishing the order_list view¶
Walk through the optimised view in full.
class OrderListView(View):
def get(self, request):
qs = (Order.objects
.filter(status='paid', customer__country='IN')
.select_related('customer')
.prefetch_related('items', 'items__product')
.order_by('-created_at')[:25])
data = [
{
'id': o.id,
'created_at': o.created_at.isoformat(),
'customer': o.customer.email,
'items': [
{'sku': i.product.sku, 'qty': i.qty}
for i in o.items.all()
],
}
for o in qs
]
return JsonResponse({'orders': data})
What hits the database:
- 1 query for the orders (with JOIN to customer via
select_related). - 1 query for items (prefetched by IN clause on order IDs).
- 1 query for products (prefetched by IN clause on item product IDs).
Three queries, regardless of how many orders are returned. Without select_related and prefetch_related: 1 + 25 + (25 × average items per order) queries.
Where the time goes in this optimised version:
- Database queries: ~5-15 ms.
- QuerySet evaluation + Python object construction: ~3-5 ms.
- JSON serialisation: ~1-2 ms.
- Middleware on the way out: ~1 ms.
Total: ~10-25 ms. Acceptable for an interactive list endpoint. Without the optimisations, the same endpoint on the same data would run 300-1200 ms.
Operational signals¶
Healthy. Per-request query count is bounded (single digits for most views). Slow-query log is quiet. Database connection pool utilisation is below 60%.
First degrading metric. Per-request query count climbing on a specific endpoint. A new feature added a foreign-key access without select_related; the count creeps up; the latency follows.
Misleading metric. Aggregate request latency. A new feature can add an N+1 to one endpoint while aggregate latency stays flat for weeks; the affected endpoint's users feel it; the aggregate hides it. Per-endpoint p95 latency, broken out, is the truth.
Expert graph. Queries-per-request per endpoint over time. A regression spikes one cell of the matrix; investigation traces to the offending view.
Where this appears in production¶
- Instagram (Django + Cinder fork) — runs Django at massive scale; their N+1 discipline (and the audit tools they built around it) are public knowledge.
- Pinterest — Django for their web tier; their guide to Django ORM optimisation has shaped the industry's vocabulary on prefetch.
- Disqus — early Django adopter; their patterns around connection pooling and query batching are foundational.
- Eventbrite — public Django performance talks document the same select_related / prefetch_related / N+1 patterns.
- The Washington Post (Arc Publishing) — Django-powered CMS; uses Cinder-style optimisations for high-traffic article rendering.
- NASA Astrobiology Institute — uses Django for data portals; queryset optimisation in scientific data lists is a recurring concern.
- DPD (parcel delivery) — Django for tracking; per-request query counts as a deployment gate.
- Mozilla — Add-ons site and many internal tools; their docs on Django ORM and signal anti-patterns are widely referenced.
Recall / checkpoint¶
- When does a QuerySet evaluate?
- What is the difference between
select_relatedandprefetch_related? - What is the N+1 problem and how does Django reveal it?
- Why does
.update()skippost_savesignals? - What does
.count()cost on a large filtered table? - Why are signals controversial in mature Django codebases?
- How do you inspect the SQL the ORM is about to run?
Interview Q&A¶
Q1. A list endpoint that used to return in 80 ms now returns in 8 seconds. The query count is in the hundreds. Walk through diagnosis.
N+1 is the prime suspect. Use connection.queries in DEBUG or APM tracing to count queries per request. Identify which related-object access is firing a query per row. Add select_related for foreign keys, prefetch_related for reverse FKs and M2Ms. Verify the count drops to single digits. Add a regression test that asserts the query count for the endpoint. Common wrong answer to avoid: "the database is slow" — it almost never is for this symptom; the application is asking too many questions.
Q2. You add select_related('customer') to a foreign key. The endpoint speed improves but memory use climbs. Why?
select_related joins the related table, so each row carries all columns of customer whether you use them or not. For wide rows (many columns, large text fields), this multiplies the bytes per result row. Mitigation: use .only() or .defer() to restrict columns; or use prefetch_related if the related model is large and not always needed. Common wrong answer to avoid: "JOINs are always faster" — they cut query count but can multiply bytes.
Q3. The team uses post_save signals to send emails. A bulk import triggers no emails. Why?
bulk_create() (and bulk_update()) does not fire per-row signals — by design, for performance. The team's email logic is in a signal that requires Model.save() per row. The honest fix is to make the side effect explicit — a service function called from both save() paths and a bulk_create path — rather than relying on signals for primary business logic. Common wrong answer to avoid: "signals always fire" — they explicitly do not for bulk operations.
Q4. The team's pagination on a 50M-row table takes 8 seconds per page. What is the structural problem?
Django's Paginator runs COUNT(*) on the filtered queryset to know how many pages exist. On a 50M-row table with a filter, this is slower than the page fetch itself. Fix: use cursor-based pagination (DRF's CursorPagination or a custom implementation that pages on an indexed column like created_at). The trade-off is no "page 17 of 200" UI; you get "next/previous" with stable performance. Common wrong answer to avoid: "add a database index" — for COUNT(), no index helps without a precomputed counter.*
Q5. A staff developer is debugging slow requests in production. They cannot reproduce locally. Walk through tooling.
Production tools: APM (Sentry, New Relic, Datadog) instruments Django and shows per-request query counts and timings; OpenTelemetry instrumentation captures spans; structured logging records the SQL of slow queries. In staging: django-silk can profile selected requests. Locally: django-debug-toolbar is the standard; connection.queries is the cheapest signal. The pattern is to capture the SQL plus timings, replay them locally with EXPLAIN ANALYZE against production-like data. Common wrong answer to avoid: "log everything" — too costly; instrument and sample.
Q6. The team's middleware list has grown to 15 items. Requests have latency cost in middleware alone. How do you audit? Time each middleware. APM tools or custom middleware that records timestamps before and after each layer can produce a per-middleware breakdown. Common culprits: SessionMiddleware reading from a slow session store (database session vs. signed cookie vs. Redis), CSRF middleware on JSON APIs that do not need it (use exemptions), custom middleware that queries the database for every request (move to per-view decorators). The fix is to size middleware to actual need, not historical accumulation. Common wrong answer to avoid: "all middleware is necessary" — it accumulated for reasons that may no longer hold.
Operational memory¶
This chapter explained what Django actually does on each request: the WSGI hand-off, the middleware chain, the URL resolution, the lazy ORM, and the patterns that cause N+1. The important idea is that the ORM is a plan-builder, not an executor; the plan runs when iterated. The cost of forgetting this is N+1, the most common Django performance bug.
You learned to recognise the QuerySet's laziness, to apply select_related and prefetch_related correctly, to inspect emitted SQL, and to spot the patterns where the ORM emits queries you did not intend. That solves the opening failure because every list endpoint can now be audited for query count.
Carry this diagnostic forward: when a Django endpoint is slow, the first question is "how many queries per request?" The answer is usually too many; the fix is usually a prefetch.
Remember:
- A QuerySet is a plan, not a result.
- N+1 is the most common Django performance bug;
select_related/prefetch_relatedis the fix. select_relatedfor single FK;prefetch_relatedfor reverse FK and M2M..update()andbulk_create()skip signals — by design.- Per-endpoint query count is the truth; aggregate latency is the appearance.
Bridge. The internals explain what each door does. Day-to-day, you write models, views, URLs, forms, templates. The next chapter is the working developer's surface — defining models with the right field types, choosing function-based vs. class-based views, structuring URLs, and the patterns that survive the team's third year. → 02-models-views-routing-day-to-day.md