Your sitemap ping has a daily budget

Telling a search engine a page changed is a request against a metered API. Most of them allow a few hundred a day, some far fewer, and the limit is per property rather than per deploy.

Nothing about that is surprising once you read it. The surprise is what your tooling does on the deploy where you cross it.

The case this post is about#

One deploy, 301 pages, 149 of them changed:

pages                301
changed              149
engine A             sitemap accepted — no per-URL metering
engine B             daily quota 100 < 149 changed → HALT quota-exhausted

Submitting every URL on every deploy had worked for months, because every deploy changed a handful of pages and a handful is under any quota.

Then one deploy changed 149, which is the deploy that most needed indexing, and the submission that had always succeeded failed — on the day the most work shipped. This post follows those four lines, because the fix is to submit what changed rather than everything, and knowing what changed requires a baseline that the version without a quota problem never needed.

The symptom#

For months, nothing. A deploy changes two or three pages, the submission loop runs, every call succeeds, and the step is invisible in the output because successful things are.

Then you do a large release — a redesign, a bulk metadata fix, a content migration — and the same loop submits a hundred and forty-nine URLs against a hundred-a-day allowance. Forty-nine are rejected.

What happens next depends entirely on code nobody has looked at since it was written. If the loop ignores response codes, the deploy reports success and forty-nine pages were never submitted. If it raises on the first rejection, your deploy fails at the indexing step after the content is already live, which is alarming and harmless and will send someone to investigate a rollback that is not needed.

Neither outcome tells you the thing you need: which pages did not make it, and when you can send them.

There is a quieter version that shows up on small deploys. If you submit every URL on every deploy, a three-hundred-page site burns three hundred of its allowance to tell the engine about two changed pages. The quota is not exceeded, nothing fails, and you have spent the budget telling a search engine about two hundred and ninety-eight pages that did not change. The first time you need headroom, it is gone, and nothing in any log explains why.

The third version is the one that cost me the most time, because it does not look like a quota problem at all. Deploy twice in a day — a release, then a typo fix an hour later — and the second run submits against an allowance the first run already spent. The changed set is two pages. The budget says no. You now have a tiny, obviously-correct deploy failing an indexing step, which reads as a bug in the step rather than as the arithmetic of a shared daily counter. I went looking at the submission code. The submission code was fine; the morning had eaten the day.

And there is a fourth that only exists because quotas reset. A step that fails at 23:50 and succeeds at 00:10 is a step that passes on retry, which is the signature of a flaky test and gets treated like one — rerun it, it goes green, move on. The rerun worked for a reason that has nothing to do with your code, and the lesson the team takes away is that the indexing step is unreliable. That is how a correct, well-behaved limit turns into a step people stop reading the output of.

None of the four announce themselves as the same problem, which is why the fix has to be in the accounting rather than in any of the four places you first notice it.

Why the obvious fix failed#

The obvious fix is to catch the error and continue. Wrap the submission, swallow the rejection, let the deploy finish.

That converts a loud failure into a silent one, which is the wrong direction. The deploy now always succeeds and the indexing step means nothing — it is a loop that runs and whose outcome nobody knows. Worse, it is the kind of code that looks careful: there is error handling right there in the diff.

The second attempt is to retry on rejection. This one actively makes things worse, because a quota rejection is not a transient error. Retrying consumes nothing but time, and a backoff loop against a daily limit will spin for hours and then fail anyway.

The third is to raise the limit — request more quota, or submit via a different mechanism. Sometimes available, usually not, and it treats the symptom. A site that submits everything every time will exhaust any budget you give it; the problem is the submission policy, not the ceiling.

The actual mechanism#

Submit what changed, and treat the quota as a resource you account for rather than discover.

Keep a hash per page from the last deploy. Compare, submit only the differences. This is the single change that matters, and it converts a three-hundred-URL submission into a two-URL one on a normal deploy, which means the budget is available on the deploy that genuinely needs it.

Know the limit before you start, not after. The number is documented per platform. Compare the size of your changed set against it up front, so the decision about what happens on overflow is made before any request is sent rather than discovered at request one hundred and one.

On overflow, submit the allowance and report the remainder by name. Not "some URLs were not submitted" — the actual list, or at minimum the count and where to find them. The next deploy, or a scheduled job tomorrow, should be able to pick up exactly what was dropped.

Different platforms, different budgets, independent handling. One engine's allowance being exhausted says nothing about another's. A loop that abandons the whole indexing step because the first platform rejected is spending the second platform's budget on nothing.

The sub-case that decides whether any of this works is what exactly you hash. Hash the rendered HTML and a footer year, a nav item, or a CSS class rename makes every page on the site changed — the diff is honest and useless, and you are back to submitting three hundred URLs, now with a mechanism that insists it is being frugal. Hash only the post's own source and you miss the case where a template change genuinely altered what the crawler sees.

Neither is right in general, and the resolution is not a cleverer hash. It is to accept that a template change is a site-wide change and handle it as its own event: a flag that says this deploy touched shared chrome, submit nothing per-URL, let the sitemap carry it. A per-URL submission API is for telling an engine about content, and a site-wide re-render is not news about any single page. That distinction has to be made in the tooling, because the hash cannot make it — the hash is correct either way and the question is what it is a hash of.

The fix#

The part that makes this operable is that the baseline of page hashes has to be written after a successful deploy, not before.

Write it before and a failed deploy leaves you with a baseline claiming pages were published that were not — so the next deploy computes a smaller diff than reality and silently never submits them. That is the same class of bug as the quota failure itself: a record that claims more than happened.

Both halves in one function — submit what changed, record only what was accepted, and report the remainder rather than dropping it:

export async function submitChanged(pages: Page[], baselinePath: string, quota: number) {
  const baseline: Record<string, string> = JSON.parse(
    await Deno.readTextFile(baselinePath).catch(() => "{}"));

  const changed = pages.filter((p) => baseline[p.url] !== p.hash);
  const batch = changed.slice(0, quota);

  const accepted: Page[] = [];
  for (const p of batch) {
    const ok = await submit(p.url);
    if (ok) accepted.push(p); else break;        // quota exhausted mid-batch
  }

  // Baseline updated ONLY for pages actually accepted. Writing it for `changed`
  // would mark the unsubmitted remainder as done, and they would never be
  // submitted again — the record claiming more than happened.
  for (const p of accepted) baseline[p.url] = p.hash;
  await Deno.writeTextFile(baselinePath, JSON.stringify(baseline, null, 2));

  return { changed: changed.length, submitted: accepted.length,
           remaining: changed.length - accepted.length };
}

A deploy that changed more pages than the daily quota allows:

$ deploy
  search indexing · 441 pages · 261 changed
  engine-a: sitemap submitted
  engine-b: 100 submitted · 161 remaining (daily quota) — retried next deploy

161 remaining is the line that matters. The version that wrote the baseline up front printed 261 changed · ok, and those 161 pages were never submitted again because the record said they already had been.

The overflow behaviour deserves one deliberate decision rather than a default. There are two reasonable policies and they suit different sites. Submit the allowance and drop the rest, with a named report is right when content changes continuously, because tomorrow's budget will catch the remainder and the list tells you what to expect. Refuse to submit at all and report is right when a partial submission would be misleading — a migration where half-indexed is worse than not-yet-indexed.

What is never right is the third option that happens by accident: submit until rejected, stop, and say nothing. That is the policy you get from a loop with no quota awareness, and it is the only one that leaves you unable to reconstruct what happened.

How we proved it#

The measurement is from a single deploy and it is the clean version of the problem.

A content change touched a hundred and forty-nine pages across a three-hundred-and-one-page site. The submission step ran against two engines with different allowances:

pages                301
changed              149
engine A             sitemap accepted — no per-URL metering
engine B             daily quota 100 < 149 changed → HALT quota-exhausted

The second engine refused, said so by name, and reported that its quota resumes tomorrow. The deploy itself succeeded — the content was live — and the indexing gap was a stated fact rather than a discovery.

What made that possible was the hash baseline already being in place. The deploy knew that a hundred and forty-nine pages had changed because it had the previous deploy's hashes to compare against, which is also what let it say the number out loud. Without that it would have submitted all three hundred and one and exhausted the budget three times over, with no way to report what was missed because it never knew what was new.

Measured on the next deploy: seventeen pages changed, seventeen submitted, quota reported as 100 → 83. The budget was there because the previous deploy had not spent it on unchanged pages.

What I have not solved: the dropped forty-nine were not automatically resubmitted. The report named the condition and the resume time; picking them up was a human remembering. A scheduled catch-up job that drains a pending-submission queue is the obvious completion and I have not built it, which means the mechanism as described converts a silent loss into a visible one rather than into a handled one.

And one thing I am taking on trust: that submission actually affects crawl priority in a way that justifies the machinery. The engines say it does. I have not run a controlled comparison between a submitted and an unsubmitted change on the same site, and I would be careful about claiming the effect size — what I can defend is that the budget is real, finite, and currently not being wasted.

What this generalizes to#

Any metered external call will eventually meet a day where you need more of it than you have, and that day is always the day you did the most work.

Send only what changed, which requires keeping a record of what you sent last time and writing that record only after the send succeeded. Know the ceiling before you start so overflow is a decision rather than a discovery. And when you do overflow, name what was dropped — because the failure mode that costs you is never the rejection itself, it is a step that ran, hit a limit, and reported the same green as the day it had room to spare.