The cache that returned someone else's data

TL;DR#

A cache key must contain every dimension that changes the answer. Miss one and the cache returns a correct answer to a different question — which, when the missing dimension is the user or the tenant, is a data leak that works perfectly and silently.

Ours was a permissions lookup keyed on the role name but not the tenant. Two tenants both have a role called admin, with different permission sets. The first lookup after a deploy decided what every tenant's admins could do.

The fix is a key-construction discipline that cannot omit a dimension: build keys from a typed object, never from string concatenation at the call site.

The case this post is about#

A permissions resolver with an obvious cache:

class PermissionService(private val db: Db, private val cache: Cache) {

    suspend fun permissionsFor(user: User): Set<Permission> {
        val key = "perms:${user.role}"                 // ← missing user.tenantId
        cache.get<Set<Permission>>(key)?.let { return it }

        val perms = db.permissionsFor(user.tenantId, user.role)
        cache.put(key, perms, ttl = 10.minutes)
        return perms
    }
}

The database query is correct — it takes the tenant. Only the cache key does not, and the key is one line in the middle of an otherwise careful function.

The symptom#

A support ticket that did not read like a security incident:

"Our admins can see the Billing section. They shouldn't be able to."

Then, twenty minutes later, the inverse from a different customer:

"Our admins can't see Billing any more. They could yesterday."

Two tenants, opposite symptoms, same cause. Whichever tenant's admin was resolved first after a cache eviction set the answer for everyone.

10:02:14  tenant=acme    role=admin  → DB  → {READ, WRITE, BILLING}   cached as "perms:admin"
10:02:19  tenant=globex  role=admin  → HIT → {READ, WRITE, BILLING}   ← globex has no BILLING

Three properties made it severe:

It was a hit, not an error. Nothing failed. The cache did exactly what it was asked.

It flipped on every eviction. Which tenant "won" depended on request ordering after each 10-minute expiry, so the behaviour was intermittent and looked like two unrelated bugs.

And it was a cross-tenant permission grant — the worst available shape, because the wrong answer was more permissive for one of them.

Why the obvious fix failed#

"Add the tenant to the key." This is the fix for this instance, and we shipped it within the hour. It does nothing about the other eleven caches in the codebase, three of which turned out to have the same shape.

"Shorten the TTL." Reduces the window and not the defect. A one-minute TTL still serves wrong data for a minute, and the leak is just as real.

"Clear the cache on deploy." Addresses a trigger, not the cause. The bug fires on any eviction, not only on deploys.

"Add a test for the tenant case." We did, and writing it revealed the deeper problem: the test had to know which dimensions mattered, which is exactly the knowledge that was missing when the key was written. A test per cache is not a system.

The actual mechanism#

A cache is a memoisation of a function. The key must be the function's complete input.

permissionsFor(tenantId, role) → Set<Permission>

That function has two inputs. The key had one. So the cache was memoising a different function — permissionsFor(role) — which is not a function at all, because it maps one input to several possible correct outputs.

Three things make this failure mode common:

Keys are strings built at the call site. "perms:${user.role}" is just text. Nothing checks it against the parameters of the thing it is caching, and nothing warns that one was omitted.

The omission is invisible in review. The line looks like a cache key. Spotting the absence of tenantId requires the reviewer to independently reconstruct what the answer depends on.

And an under-specified key always hits. A key that is too broad produces wrong answers with a high hit rate; a key that is too narrow only costs performance. The dangerous direction is also the one that looks like it is working well.

A cache key is the identity of the question. Omit a dimension and you have cached the answer to a question nobody asked — and if that dimension is a security boundary, the cache is an authorisation bypass with a good hit rate.

The fix#

Make the key impossible to under-specify: a typed key object, constructed from the same parameters the function takes.

// one type per cached function, with the function's full input as its fields
sealed interface CacheKey {
    val namespace: String
    fun render(): String

    data class Permissions(val tenantId: TenantId, val role: Role) : CacheKey {
        override val namespace = "perms"
        override fun render() = "$namespace:v1:${tenantId.value}:${role.value}"
    }
}

class PermissionService(private val db: Db, private val cache: Cache) {

    suspend fun permissionsFor(user: User): Set<Permission> {
        // the key type's constructor REQUIRES both dimensions
        val key = CacheKey.Permissions(tenantId = user.tenantId, role = user.role)
        cache.get<Set<Permission>>(key)?.let { return it }

        val perms = db.permissionsFor(user.tenantId, user.role)
        cache.put(key, perms, ttl = 10.minutes)
        return perms
    }
}

// the cache only accepts typed keys — string keys are no longer expressible
interface Cache {
    suspend fun <T> get(key: CacheKey): T?
    suspend fun <T> put(key: CacheKey, value: T, ttl: Duration)
}

Four decisions:

The key is a data class with required fields. Omitting the tenant is now a compile error rather than a missing substring. This is the whole fix — the rest is supporting it.

Cache takes CacheKey, not String. Removing the string overload is what prevents the next ad-hoc key. Keeping it would leave the old path available, and the old path is the bug.

A v1 version segment in the rendered key. When the shape of the cached value changes, bump it rather than relying on a deploy-time flush — so old entries cannot be deserialised into a new type.

And TenantId/Role as value classes, not String, so the two cannot be passed in the wrong order:

@JvmInline value class TenantId(val value: String)
@JvmInline value class Role(val value: String)

Then an audit, because this was never only one cache:

# find every remaining ad-hoc key construction
$ grep -rn 'cache\.\(get\|put\)(".*\${' --include='*.kt' .

That search found eleven call sites. Three were missing a dimension:

SessionCache      key="session:${token}"             ✓ token is globally unique
FeatureFlagCache  key="flags:${userId}"              ✗ missing environment
SearchCache       key="search:${query}"              ✗ missing tenantId AND locale
ReportCache       key="report:${reportId}"           ✗ missing the requesting user's role

SearchCache was the second cross-tenant leak, found by audit rather than by a support ticket.

And a review checklist item, because the typed key makes omission harder and not impossible:

For every cache: list what the answer depends on. Is each one in the key type?
  - who is asking (user, role, tenant)
  - what is being asked (the parameters)
  - under what conditions (locale, environment, feature flag, schema version)

How we proved it#

The leak, reproduced deterministically — which turned two confusing tickets into one bug:

@Test
fun twoTenantsWithTheSameRoleNameDoNotShareCachedPermissions() = runTest {
    db.setPermissions(TenantId("acme"),   Role("admin"), setOf(READ, WRITE, BILLING))
    db.setPermissions(TenantId("globex"), Role("admin"), setOf(READ, WRITE))

    val acme   = service.permissionsFor(User(TenantId("acme"),   Role("admin")))
    val globex = service.permissionsFor(User(TenantId("globex"), Role("admin")))

    assertEquals(setOf(READ, WRITE, BILLING), acme)
    assertEquals(setOf(READ, WRITE), globex)          // ← was {READ, WRITE, BILLING}
}

before:  FAILED — globex got BILLING
after:   PASSED

Ordering matters in that test and the fix makes it irrelevant; before the fix, reversing the two calls reverses which tenant is wrong, which is the intermittency reproduced.

That the string path is gone:

$ grep -rn 'cache\.get("' --include='*.kt' . | wc -l
0
$ # and the compiler enforces it
$ cache.get<Set<Permission>>("perms:admin")
e: type mismatch: inferred type is String but CacheKey was expected

The audit results, as a count rather than a claim:

cache call sites:                    11
keys missing a dimension:             3
of those, cross-tenant leaks:         2   (permissions, search)
of those, found by a user report:     1

One found by a customer, one found by us. The ratio is the argument for auditing rather than fixing the reported instance.

And the exposure window, from the logs, because an honest incident writeup includes it:

cache hits on under-specified "perms:*" keys, 30 days before the fix
  total:                  1,284,117
  cross-tenant mismatches:   41,208   (3.2%)
  of which more permissive:  19,744

Nineteen thousand requests served a more permissive permission set than the tenant was entitled to. We disclosed to the affected customers; no evidence of the extra permission being exercised was found in the audit log, which is a mitigating fact and not an excuse.

One limit: the 3.2% is reconstructed by replaying logged (tenant, role) pairs against the cache's observed key sequence, so it is an estimate of mismatches rather than a per-request record. The direction and magnitude are reliable; the exact number is not.

What this generalizes to#

A cache key is the identity of a question, and every dimension must be in it. Who is asking, what they asked, and under what conditions. Memoisation, HTTP caching, CDN keys, derived tables, a @lru_cache — the same rule, and the same failure when a dimension is implicit.

The dangerous direction of a key error is the one that looks healthy. Too narrow costs performance and is obvious. Too broad serves wrong answers with a high hit rate. A suspiciously good hit rate on a tenant-scoped cache is worth a second look.

Make omission a compile error. A typed key whose constructor requires every dimension removes the class of bug rather than the instance. Any time correctness depends on remembering to include something, there is a type that could have required it.

Audit the pattern after fixing the instance. The reported bug was one of three, and one of the other two was also a cross-tenant leak. A grep for the shape takes minutes and is the difference between fixing a ticket and fixing a defect class.

Related#

• remember is not a cache, and the key is the whole contract — the same keying discipline, client-side
• Idempotency is a product decision before it is a technical one — another identity-definition problem
• Logs are for the questions you have not thought of yet — how the exposure window was reconstructed

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.