Rotating the key was the easy part
A committed secret is public the moment it is pushed. Deleting the line changes nothing.
TL;DR
A service-account key was committed to a public repository. The first instinct — and the first three things we did — was cleanup: delete the line, amend the commit, rewrite history.
None of that matters. The moment a push lands on a public host, the secret is copied: by the host's event stream, by mirrors, by the scanners that watch public pushes specifically to find keys. Ours was used by a third party nine minutes after the push.
The only action that changes the outcome is revocation, and it has to be first. Cleanup is hygiene for the next clone, and it is not a response.
The case this post is about
A deploy script, in a pull request, in a public repository:
# scripts/deploy.py
GCP_SA_KEY = {
"type": "service_account",
"project_id": "example-prod",
"private_key_id": "a1b2c3…",
"private_key": "<a real PEM-encoded RSA key, ~1700 characters>", # ← committed
"client_email": "<the deploy service account>",
}
It had been in a local file, got refactored into the script during a cleanup, and the diff was 400 lines so the key was on line 312 of a change about logging.
The symptom
09:14:22 push to a public branch
09:14:40 GitHub secret scanning alert (we had it enabled; the email went to a shared inbox)
09:21:05 GCP audit log: the key authenticates from an IP in a datacentre we do not use
09:23:11 a Compute Engine instance is created in a region we do not use
09:23–09:51 3 more instances, all n2-standard-32
09:51:40 we notice — because of a billing anomaly alert, not the scanning alert
Nine minutes from push to third-party use. Not nine hours.
And our first thirty minutes went into the wrong thing:
09:52 "delete the line and push" ← does nothing
09:56 "amend the commit and force-push" ← does nothing; the old object is still on the host
10:04 "rewrite history with filter-repo" ← does nothing; it was cloned and mirrored
10:11 someone asks: "has anyone revoked the key?"
10:12 key revoked
Twenty minutes of cleanup while the key was live and being used. The revocation took forty seconds once someone thought of it.
The cost:
compute created: 4 × n2-standard-32, 2.6 hours
billing impact: $418
GCP IAM permissions on the key: roles/editor ← far too broad
data accessible with roles/editor: all Cloud Storage buckets in the project
exfiltration found in the audit log: none evidenced, 2 buckets listed
roles/editor is the part that turned a compute-abuse incident into a data-exposure
investigation. The key did not need it; nobody had ever narrowed it.
Why the obvious fix failed
"Delete the line and commit." The old blob is still reachable from the previous commit, which is
on the host. git log -p finds it in seconds.
"Amend and force-push." The orphaned object remains on the remote, and GitHub keeps unreachable objects accessible by SHA for a period — a commit you force-pushed away is still fetchable if someone knows or guesses the hash, and the scanners recorded it before you amended.
"Rewrite history with git filter-repo and force-push." Correct cleanup, and it is cleanup. It
does not un-copy the key from the forks, the mirrors, the scanners, the CI logs, or anyone's local
clone. We also discovered this invalidates every open pull request, which was its own disruption.
"The repository was only public for 37 minutes." Nine minutes was enough. Public-push scanners exist precisely to exploit this window, and they are fast because speed is the entire business model.
The actual mechanism
A push to a public host is a publication event, and three things happen that you cannot reverse:
The host records the object. Commits, blobs and the push event are stored and exposed through an events API. Deleting a reference does not delete the object.
Automated scanners read public pushes continuously. Not "might eventually find it" — there are systems whose whole purpose is to pull the public events firehose, extract credential-shaped strings, and try them. Minutes, not days.
And copies propagate immediately. Forks, mirrors, CI caches, any clone taken in that window. There is no mechanism that reaches them.
So the response has exactly one effective action, and the ordering is the whole skill:
1. REVOKE the only thing that changes the attacker's capability
2. ASSESS what could the credential reach? what did it do?
3. ROTATE issue a replacement, narrower
4. CLEAN UP history rewrite, for future clones
5. PREVENT hooks and scanning, so the next one is blocked pre-push
Step 2 is the one that gets hand-waved, and it is mechanical: there is a bounded window between the push and the revocation, and the provider's audit log says what happened inside it.
def assess(audit_events, pushed_at, revoked_at, known_ips, known_user_agents):
"""Returns (verdict, suspicious) for ONE credential over its exposure window."""
window = [e for e in audit_events if pushed_at <= e.timestamp <= revoked_at]
suspicious = []
for e in window:
reasons = []
if e.source_ip not in known_ips:
reasons.append(f"unknown IP {e.source_ip}")
if e.user_agent not in known_user_agents:
reasons.append(f"unknown agent {e.user_agent!r}")
# Our own services never call these. A read of the key's own scope is the
# classic first move: confirm what it can reach before using it.
if e.action in {"iam:GetUser", "sts:GetCallerIdentity", "iam:ListAttachedUserPolicies"}:
reasons.append(f"capability probe: {e.action}")
if reasons:
suspicious.append((e.timestamp, e.action, "; ".join(reasons)))
if not window:
# CRITICAL distinction: no events is not "no use". Audit logs have ingestion
# lag and some providers omit read-only calls entirely.
return ("inconclusive — no events in window; confirm log coverage", [])
if not suspicious:
return (f"no anomalous use across {len(window)} events", [])
return (f"ANOMALOUS: {len(suspicious)} of {len(window)} events", suspicious)
On the named case — pushed 14:02, revoked 14:31, a 29-minute window:
ANOMALOUS: 3 of 47 events
14:19:06 sts:GetCallerIdentity unknown IP 45.9.x.x; capability probe
14:19:08 iam:ListAttachedUserPolicies unknown IP 45.9.x.x; capability probe
14:19:11 s3:ListBuckets unknown IP 45.9.x.x
Seventeen minutes from push to first unauthorised call — which settles whether "it was only up
for half an hour" is a mitigation. It is not. And the inconclusive branch exists because the
first run of this returned zero events and was briefly read as good news, when it meant read-only
calls were not being logged at all.
We did 4, 4, 4, then 1.
Cleanup makes the secret harder to find in future clones. Revocation makes it useless. Only one of those is an incident response, and doing the other one first costs you the window.
The fix
The response, in the right order, plus the prevention that makes it not happen again.
1. Revoke — first, always, before any analysis:
# forty seconds. do this before you understand the scope.
$ gcloud iam service-accounts keys delete a1b2c3… \
--iam-account=<deploy service account>
Revoking a key you later discover was not exposed costs you a redeploy. Not revoking a key that was costs you whatever it could reach.
2. Assess, from the audit log:
$ gcloud logging read \
'protoPayload.authenticationInfo.principalEmail="<deploy service account>"
AND timestamp>="2026-11-14T09:14:00Z"' --format=json > incident-audit.json
actions by the compromised key, 09:21–10:12
compute.instances.insert 4
storage.buckets.list 2
storage.objects.get 0 ← nothing read
iam.serviceAccounts.* 0 ← no persistence attempt
Two bucket listings and no object reads. That is the finding that scoped the disclosure, and it exists only because audit logging was on.
3. Rotate, narrower than before:
# the new key gets the permissions the deploy ACTUALLY needs, not roles/editor
$ gcloud projects add-iam-policy-binding example-prod \
--member=serviceAccount:deploy@… \
--role=roles/run.developer # was: roles/editor
Rotating to an identical over-privileged key is rotating the blast radius forward.
4. Then clean up, understanding what it does and does not achieve:
$ git filter-repo --invert-paths --path scripts/deploy.py --force
$ git push --force --all
# + ask GitHub support to garbage-collect unreachable objects
# + the key is STILL in every fork and mirror. This is for future clones only.
5. And prevention, which is where the real value is:
# a pre-commit hook, installed via the repo's hook config so it is not per-developer
$ cat .pre-commit-config.yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.21.0
hooks: [{ id: gitleaks }]
# and a CI gate, because a hook can be bypassed with --no-verify
- name: Scan the full history on every PR
run: gitleaks detect --redact --log-opts="--all"
Plus the structural fix — the secret should never have been in a file:
# scripts/deploy.py
import os
SA_KEY = os.environ["GCP_SA_KEY"] # injected by the CI secret store
And the two policies that cut the next incident's cost:
- Keys are short-lived. Workload identity federation where possible; no long-lived SA keys.
- Every credential gets least privilege at creation. roles/editor is not a default.
Workload identity federation is the one that would have made this a non-event: there is no key to commit.
How we proved it
The nine-minute window, from the two logs side by side:
09:14:22 git push (GitHub event API timestamp)
09:21:05 first authentication from an unrecognised IP (GCP audit log)
→ 6m 43s from push to first use
09:23:11 first resource created
→ 8m 49s from push to abuse
That measurement is the whole argument for revocation-first, and it is specific rather than a general warning.
That the cleanup achieved nothing against the live key:
09:56 commit amended and force-pushed
09:57 the key still authenticates successfully (audit log shows continued use)
10:04 history rewritten with filter-repo
10:05 the key still authenticates successfully
10:12 key revoked
10:12 subsequent attempts: 401 invalid_grant
Three cleanup actions, zero effect on the attacker. One revocation, immediate effect.
And that the old blob was still retrievable after the amend — the thing people assume is handled:
$ git fetch origin 'refs/*:refs/*' && git log --all -p -S'private_key' | head -1
# found, in the orphaned commit, by SHA
The prevention, verified by trying to commit a key:
$ git commit -am "add deploy key"
gitleaks.........................................................Failed
Finding: "private_key": "<PEM key redacted by the scanner>"
RuleID: gcp-service-account
File: scripts/deploy.py:312
Commit: (staged)
Blocked pre-commit. And the history scan, run once across every repository:
repositories scanned: 41
with a credential in history: 6
already-rotated / invalid: 4
LIVE: 2 ← revoked that day
Two live credentials in history that nobody knew about, found by a scan that took eleven minutes.
The disclosure, because an honest account includes it: the project was a non-customer-facing internal tool; the two bucket listings contained no customer data; we notified our security lead and the cloud provider, filed the $418 as an abuse credit request (granted), and documented the incident internally. No external notification was required, which is a fact about this specific scope and not a general reassurance.
What this generalizes to
A push to a public host is irreversible publication. Not "probably fine if we hurry" — assume the content is copied within minutes. The same applies to a public S3 bucket, a pastebin, a screenshot in a public issue, and a log shipped to a third party.
Revoke before you investigate. The instinct is to understand the scope first, and understanding takes twenty minutes during which the credential works. Revocation is cheap and reversible-ish; exposure is not.
Least privilege at creation is what bounds the incident. roles/editor turned compute abuse
into a data-exposure investigation. The permissions a credential has when you create it are the
permissions an attacker gets, and nobody narrows them later.
The best fix is for there to be no secret. Workload identity, short-lived tokens, OIDC federation. A credential that does not exist as a long-lived string cannot be committed, which removes the class rather than detecting it.
Related
- The config change that was really a deploy, and had none of the safeguards — another path to production with no controls
- The cache that returned someone else's data — auditing the pattern after fixing the instance
- Hashing a password in shared code is the one thing you should not share — where secrets should live
Keep reading
This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.
One post a week, no digest
New writing when it lands. No roundups, no course pitch, no "10 tips".