We shipped it, measured it, and deleted it eleven weeks later

TL;DR#

We built a saved-searches feature over five weeks. It shipped, 0.8% of users created one, and we deleted it eleven weeks later — which is unusual, because features almost never get deleted.

What made deletion possible was a decision made before building: a written success threshold (8% adoption at 8 weeks) and a pre-committed action if it was missed. Without that, the conversation afterwards is anecdote against anecdote, and the feature always wins — because someone can always name a user who likes it.

The threshold is cheap to write and it is the only thing that makes "we were wrong" an available conclusion.

The case this post is about#

A saved-searches feature: let users name and store a filter combination, surface it on the search screen.

The spec, which included something most specs do not:

## Success criteria (agreed before build, 2026-06-02)
- PRIMARY: ≥8% of weekly-active users create at least one saved search within 8 weeks of GA.
- SECONDARY: ≥40% of creators use a saved search again within 14 days.
- Rationale for 8%: the three existing power-user features sit at 11%, 9% and 6% adoption.
  Below 6% we have historically not been able to justify the maintenance.

## If the primary is missed
Remove the feature. Specifically: hide the entry point, stop writing new saved searches,
export existing ones to the user's account export, delete the code after a 4-week
notice period. Owner: the team that built it.

Five weeks of work, two engineers and a designer.

The symptom#

Adoption, measured weekly against the threshold:

week   WAU creating ≥1 saved search
1                 0.4%
2                 0.6%
4                 0.7%
6                 0.8%
8                 0.8%        threshold: 8.0%

A tenth of the target, flat from week 4.

The secondary metric was actually fine, which is the interesting part:

creators who used a saved search again within 14 days:  61%   (threshold 40%)

The people who found it liked it. There were 340 of them out of 42,000 weekly actives.

And the maintenance cost had already begun:

weeks 1–11
  bugs filed against saved searches:        7
  engineer-days on those bugs:             4.5
  the feature appeared in:
    the search screen's state model (3 fields)
    2 database tables
    the account-export job
    the onboarding tour
    1 migration in an unrelated refactor

Four and a half engineer-days of maintenance for 340 users, plus a permanent presence in five places that every future change to search had to account for.

Why the obvious fix failed#

"Promote it harder." We did, in week 5: a tooltip on the search screen and one entry in the release notes. Adoption went from 0.7% to 0.8%. Discovery was not the constraint — the tooltip was seen by 31,000 users and converted 140.

"Make it easier to use." The secondary metric says the people who used it had no trouble. A usability fix addresses a problem the data says we did not have.

"Give it more time." Flat from week 4 to week 8. The argument for more time is always available and, with a flat curve, is not supported by anything.

"Keep it — it's already built, deleting is waste." The most persuasive argument in the room and the one the pre-commitment exists to answer. The build cost is sunk either way; the only live question is the future maintenance cost against the future value. Framing it as "wasting the work" conflates those.

The actual mechanism#

Three forces keep an unsuccessful feature alive, and all three are about the absence of a threshold rather than about the feature.

Sunk cost reads as justification. Five weeks of work feels like a reason to keep something. It is a reason the decision was expensive, not a reason the decision was right.

Anecdote beats aggregate in a meeting. "340 people use it weekly and 61% come back" is true and compelling. "That is 0.8% against an 8% bar" requires the bar to exist — and without one, the specific user story wins, every time.

And nobody owns removal. Building has a champion; deleting has nobody. The default action in the absence of a decision is to keep, so a feature survives by inertia rather than by argument.

A pre-committed threshold changes the default:

without a threshold:   default = keep.  Deleting requires someone to argue for it.
with a threshold:      default = the pre-agreed action. Keeping requires a new argument.

Written as code, the whole idea is that the decision is a lookup rather than a discussion — and that the function refuses to run at all if the bar was not set in advance:

from dataclasses import dataclass
from datetime import date

@dataclass(frozen=True)
class Commitment:
    bar: float            # adoption fraction that counts as success
    review_on: date
    if_below: str         # the action, agreed BEFORE launch
    committed_on: date

def evaluate(name, measured, active_users, c, launched_on, today):
    # A bar agreed after launch is not a pre-commitment — it is a number chosen while
    # already knowing the answer. Refuse, rather than lending it the authority of this
    # process; that refusal is the single most important branch here.
    if c.committed_on > launched_on:
        raise ValueError(
            f"{name}: bar set {(c.committed_on - launched_on).days}d AFTER launch — "
            "not a pre-commitment, and not evaluable by this rule")

    if today < c.review_on:
        return f"{name}: too early, review on {c.review_on}"

    adoption = measured / active_users
    if adoption >= c.bar:
        return f"{name}: KEEP — {adoption:.1%} at or above {c.bar:.0%}"

    # Note what is NOT consulted: build cost, retention among the users who DID adopt,
    # and how good the anecdote is. All three are real, and all three are why the
    # conversation never reached a conclusion.
    return (f"{name}: {c.if_below.upper()} — {adoption:.1%} against a {c.bar:.0%} bar "
            f"({measured:,} of {active_users:,})")

On the named case:

>>> evaluate("saved-filters", measured=340, active_users=42_500,
...          c=Commitment(bar=0.08, review_on=date(2026, 9, 1),
...                       if_below="remove", committed_on=date(2026, 5, 2)),
...          launched_on=date(2026, 5, 12), today=date(2026, 9, 3))

'saved-filters: REMOVE — 0.8% against an 8% bar (340 of 42,500)'

The 61% retention among those 340 does not appear anywhere in that output, and that is deliberate. It was the strongest argument in the room and it answers a different question: those users like it. The bar asked whether enough people adopted it, and 0.8% is an answer, not an opinion.

The value of a pre-committed threshold is not the number. It is that it makes "we were wrong" an available conclusion — reachable without anyone having to advocate for it against the people who built the thing.

The fix#

The threshold was the fix, written five weeks before it was needed. What remained was executing it well.

1. Honour the pre-commitment, and say so explicitly:

## Decision, 2026-08-18
Primary criterion missed (0.8% vs 8.0%). Executing the pre-agreed removal.
This is not a judgement on the implementation — the secondary metric (61% repeat
use) suggests it works well for the people who want it. There are not enough of them.

Separating "the feature is bad" from "the demand is small" is what let the team own the decision without it reading as a failure of their work.

2. Remove the entry point first, keep the data:

// week 1 of removal: no new saved searches, existing ones still readable
if (flags.isEnabled("saved_searches_create")) { /* hidden */ }

3. Notify the 340 users with a real export path:

subject: Saved searches is being retired on 15 September

You have 4 saved searches. They will keep working until 15 September, and you can
download them now — or copy any of them into a bookmark with one tap.
[ Download my saved searches ]  [ Turn each into a bookmark ]

The "turn into a bookmark" option is the part worth copying: an existing feature covered most of the need, and 212 of the 340 took it.

4. Then delete, with the evidence in the commit message:

refactor(search): remove saved searches

Adoption 0.8% of WAU at 8 weeks against a pre-agreed 8% threshold (spec 2026-06-02).
Secondary metric met (61% repeat use among creators) — the feature worked, the
demand did not exist.

Users notified 2026-08-20; 4-week notice; 212 of 340 migrated to bookmarks.
Removes: 2 tables, 3 state fields, 1,840 lines, 7 open bugs.

5. And the practice, generalised into the spec template:

## Success criteria          (required; a spec without this is not ready)
- PRIMARY: <metric> ≥ <threshold> by <date>
- Rationale for the threshold: <comparable features, or the cost it must justify>
## If the primary is missed
- <specific action>, owned by <team>
## Review date
- <date>, on the calendar now

The calendar entry matters more than it looks. A review date that exists as an invite happens; one that exists as an intention does not.

How we proved it#

That adoption was genuinely flat, not still growing:

week-over-week growth in creators
  w1→w2  +52%
  w2→w4  +18%
  w4→w6   +6%
  w6→w8   +1%

A decaying growth rate at 0.8% is not an early-adoption curve, which is the thing "give it more time" implicitly claims.

That promotion was not the constraint:

week-5 tooltip
  users shown:       31,400
  tapped:             1,180   (3.8%)
  created a search:     140   (0.45% of shown)

The cost of keeping it, projected — which is the number the decision actually turns on:

observed maintenance, 11 weeks:              4.5 engineer-days
projected annual:                           ~21 engineer-days
plus: present in 5 subsystems, so every search refactor carries it
value: 340 users × 61% repeat use

Twenty-one engineer-days a year for 340 users was the comparison, and it is only available because someone wrote down what the bar was.

And the outcome after deletion, which is the part I did not expect:

search screen state fields:        11 → 8
search-related bugs, next quarter:  14 → 6
the search refactor planned for Q4 shipped 1 week early

The counterfactual check, because the honest question is whether we deleted something valuable:

support tickets mentioning saved searches, 12 weeks after removal:  9
of those, asking for it back:                                       4

Four requests in three months. That is the measured answer to "were we wrong to delete it", and it is a much better answer than the pre-deletion argument could have produced.

One limit: the 8% threshold was derived from three comparable features in our own product, which is a small and self-referential basis. A different bar would have produced a different decision, and I cannot claim 8% was the correct number — only that having a number, agreed in advance, is what made the decision possible at all.

What this generalizes to#

Pre-commit the threshold and the action, before building. Not to be rigid, but to make the unfavourable conclusion reachable. A decision rule written before the result is known is the only one not contaminated by the result.

Sunk cost is not a reason; future cost is. The build is spent either way. The live comparison is always future maintenance against future value, and framing removal as "wasting the work" is the cognitive error the threshold exists to interrupt.

Separate "it works" from "it is wanted". Our secondary metric was met: the implementation was good. Conflating the quality of the work with the size of the demand makes a deletion feel like a verdict on the team, which is exactly what makes it hard to do.

Put the review on the calendar. The threshold without a date is an intention. Fifteen minutes in a calendar eleven weeks out is what converts it into a decision that actually gets made.

Related#

• Engagement went up, and it was because the app got slower — choosing the metric itself
• The most valuable pull request that quarter deleted 4,000 lines — removing code nobody reaches
• An estimate is a distribution, and you reported the mode — decisions made before the outcome is known

Keep reading#

This is part of 100 practices of senior engineers — the habits that separate five years of experience from one year five times. Follow MobileByteSensei for the rest.