dhaga.blog
Engineering

Isolating tenants on serverless Postgres

Row-level security is easy to turn on and easy to get catastrophically wrong. How a connection pooler almost leaked one customer's data into another's in Dhaga Cloud, and the transaction-scoped design that makes it impossible on either pooling mode.

The short version

Dhaga Cloud is multi-tenant: many customers' relationship graphs live in one Postgres database, kept apart by Postgres row-level security (RLS). RLS is the right tool — but how you tell the database "this request is user X" decides whether a serverless connection pooler can leak that identity onto another tenant's query. Our first version scoped it with a session-level setting, which collides violently with transaction pooling: the setting can ride onto the wrong connection, and one tenant can render another tenant's data.

We caught it in end-to-end testing, not production, and shipped a two-part fix — a fail-loud boot guard against the unsafe pooler, plus a RESET ALL scrub on every connection release. Then we replaced both with something simpler: scope each request inside one transaction, with a transaction-local setting that Postgres discards automatically at COMMIT. No boot guard, no scrub, and correct on either pooling mode. This post is the whole trap and both fixes.

The rest of this post is the deep dive. It's a good read for anyone running RLS behind a connection pooler on serverless — the failure mode is subtle and the blast radius is "cross-tenant data exposure." File paths refer to the real code.


The setup: RLS, and how a request says who it is

The multi-tenant machinery lives in packages/ee/src/db/ — the Dhaga Cloud package. A single-tenant deployment needs none of it.

Every tenant-scoped table carries a user_id, and an RLS policy says you can only see rows where user_id matches the current request's user. The policy DDL expresses "the current request's user" as a Postgres session setting:

-- a row is visible if you're the admin, or it's yours
USING (
  current_setting('app.bypass_rls', true) = 'true'
  OR user_id = current_setting('app.current_user_id', true)
)

So before running a tenant's queries, we set that variable — set_config('app.current_user_id', <the user>, …) — and Postgres does the rest. Admin/webhook code paths set app.bypass_rls instead. Clean, database-enforced isolation: even a bug in application code can't return another tenant's rows, because the database filters them.

That's the theory. Now the trap.

The trap: session state and transaction pooling don't mix

Serverless functions can't hold long-lived Postgres connections — there can be thousands of concurrent function invocations and Postgres tops out at a few hundred connections. So you put a pooler in front. Supabase (our managed Postgres) offers two modes, and the difference is the whole story:

Pooler modePortHow it hands out backends
Session5432One backend per client connection, for the life of that connection
Transaction6543A backend is borrowed per transaction and can change between queries

Our first isolation design depended on a session-level setting — app.current_user_id persisted on a connection across queries. Transaction pooling breaks that assumption in two distinct, both-bad ways:

  1. Silent under-fetch. You set_config on one backend, then your next query runs on a different backend that never got the setting. current_setting(…, true) returns empty, the RLS policy matches nothing, and the tenant sees an empty graph. Annoying, but fail-safe.
  2. Cross-tenant leak. The genuinely dangerous one. A backend that still carries tenant A's app.current_user_id gets handed to a request for tenant B. Now tenant B's queries run as tenant A — B renders A's data. That's not a bug, that's an incident.

We proved case (1)/(2) live in an end-to-end test on 2026-07-16: intermittent zero-row renders, and the session setting bleeding across pooled backends. The scariest part is how quiet it is — most requests are fine, because most of the time the backend you get happens to be yours.

The first fix: forbid the unsafe mode, and scrub on reuse

Our first version attacked the trap from two directions at once.

Refuse to boot in the unsafe mode. The strongest form of a safety rule is to make the unsafe configuration impossible to run. So a fail-loud boot guard in bootstrap.ts checked the connection string at startup and threw if it looked like a transaction pooler — DATABASE_URL had to point at the session pooler (port 5432). A direct application of our "fail loud" principle: a misconfiguration that could expose data should crash the deploy, not degrade silently in production.

Scrub every connection before reuse. Session mode then surfaced a second, subtler problem. Under session pooling we run small, per-role connection pools — a tenant pool (default max 3, via DB_POOL_MAX_TENANT) and a core pool (default max 2), so a single serverless instance holds at most 5 of Supabase's shared connection slots. One instance hoarding all the slots is its own outage, so those maxes are deliberately low and must not be raised blindly. Small pools mean connections get reused across requests — and a reused connection still carried the app.current_user_id (or app.bypass_rls) from whoever used it last. If request B checked out the connection request A just returned, B inherited A's identity. So on release we ran RESET ALL to wipe every session variable before handing the connection back, destroying it only if the reset itself failed.

This worked — connections were reused (fast) and carried no identity across checkouts (safe). But it left two things we weren't happy about. The whole system was locked to the session pooler — the boot guard existed precisely to enforce that — so we couldn't move to Supabase's transaction pooler later without a rewrite. And correctness leaned on remembering to scrub every connection on every path. Both are the kind of load-bearing discipline that quietly rots. So we replaced them with a design that needs neither.

The better fix: scope inside one transaction

The root cause of the entire trap is that app.current_user_id was a session-level setting that lived on a connection between transactions — which is exactly the window a transaction pooler reassigns backends in. Take that window away and the whole class of problem disappears.

So each unit of tenant work now runs inside a single transaction, and the setting is made transaction-local:

BEGIN;
SELECT set_config('app.current_user_id', $1, true);  -- is_local = true
-- … the tenant's queries run here, RLS-scoped …
COMMIT;  -- Postgres discards the setting automatically, right here

That third true argument is the whole trick: a transaction-local setting is discarded the instant its transaction ends (scoped-db.ts). One change, and both earlier fixes fall away:

  • No RESET ALL on release. There is nothing left to reset — the setting is already gone at COMMIT. Release just returns the physical connection to the pool (releaseScoped), so we keep connection reuse (no handshake per request) with no scrub discipline to get wrong. A connection that won't release cleanly is still destroyed rather than returned dirty.
  • No boot guard, and no pooler lock-in. Because the scope lives entirely inside one transaction that sets its own setting first, a transaction pooler never gets the chance to run a query unscoped, or to carry the setting onto a backend it later hands to someone else. The same code is correct on both pooling modes — the session pooler (5432) and the transaction pooler (6543). Flipping between them becomes a DATABASE_URL change, no code change, which is what lets us move to the transaction pooler at Supabase Pro without touching the isolation layer. The old boot guard (and its DHAGA_ALLOW_TRANSACTION_POOLER escape hatch) was retired as obsolete.

One honest caveat: the transaction-pooler (6543) path is designed-correct and follows directly from the reasoning above, but we have not yet run it against a live 6543 pooler — that verification is still pending before we flip DATABASE_URL at Pro. The session pooler (5432) is what serves production today.

Proving it stays fixed

A subtle isolation guarantee that isn't tested is a guarantee that will regress. The isolation layer is covered from two angles:

  • Unit tests (a fake pg client under real drizzle) assert that a tenant scope wraps its work in BEGIN … COMMIT, sets the GUC transaction-local (set_config(…, true) as a bound parameter, never interpolated), rolls back on error, and issues no RESET ALL on release. They fail the moment the scope stops being transaction-local or a session reset creeps back in.
  • A real-Postgres integration test (skipped when no DATABASE_URL is present) reuses a single backend across two tenants and asserts the second checkout carries no user_id/bypass from the first and that RLS isolates the two end-to-end — the property transaction-local scoping exists to guarantee. Verified green against real Postgres on the session pooler; the transaction pooler runs the same code and is the pending live check above.

The guarantee is now a property of when the setting exists — one transaction, no longer — rather than of a scrub we have to remember to run.

A footnote that cost real milliseconds: put the function next to the database

While chasing this we found an orthogonal but expensive issue: the Supabase database was in Sydney (ap-southeast-2) and the Vercel function was running in US-East. Every query paid a ~200ms cross-Pacific round trip, and a page doing a dozen queries paid it a dozen times — a big chunk of an 11-second render. Pinning the function to the database's region is the fix. It's a reminder that in serverless, where your compute runs relative to your data is a first-class performance decision, not an afterthought.

The takeaways

  1. RLS is enforced by the database — but scoped by your connection. The moment a pooler can move your queries to a different backend, or hand your backend to someone else, your isolation model has a hole. Understand your pooler's mode before you trust connection-scoped RLS.
  2. Prefer a design with no unsafe mode over a guard against it. Our first fix was a fail-loud guard forbidding the transaction pooler; the better fix made the pooler mode not matter. A configuration you can't get wrong beats a loud error when you do.
  3. Scope to a transaction, not a session. A transaction-local setting (set_config(…, true)) is discarded at COMMIT for free — no RESET ALL, no scrub-on-release to forget, and it's correct under both session and transaction pooling. Reaching for a session variable across the pooler was the original mistake.
  4. Test the isolation itself. A real two-tenant integration test that asserts "B cannot see A" is the only thing that keeps a silent leak from coming back.
  5. Co-locate compute and data. Cross-region round trips are invisible in dev and painful in prod.

Every account on Dhaga Cloud is isolated by these policies. A single-tenant instance needs none of this machinery — self-hosted deployments are available to enterprise customers on request. See Self-hosting.

Share

Discussion

On this page