An agent that may only work on the front of the queue

A drive is given a queue of work and a goal. It reads the queue, identifies the item that most obviously advances the goal, and starts there.

Four items were skipped, and nothing recorded that they were skipped.

The symptom#

A run that makes real progress on something while leaving a backlog it has silently reclassified as optional. The work done is good work; the items passed over are not failures, not deferrals, not anything — they were simply not chosen, and not choosing leaves no trace.

The second thing is why the skip happens, which is not laziness. The skipped items usually look like setup: scaffolding, registration, a configuration step, something that produces no visible result. Reaching past them to the item that visibly advances the goal is exactly what a system optimising for the goal should do, and it is correct right up until one of those setup items was a prerequisite.

The third is the shape of the eventual failure. A later step fails for a reason that makes no sense — a missing registration, an absent directory, a capability that was never initialised — and tracing it back leads to an item that was in the queue the whole time and was never dispatched.

The fourth is that the queue still looks reasonable afterwards. The skipped items are still in it, still pending, still perfectly valid. Nothing is corrupt; the run simply did not do them, and the queue has no notion of having been passed over.

That is the property that makes this different from an ordinary missed step: the state is consistent and the backlog is intact. There is no anomaly to detect, only an absence of progress on items nobody is tracking individually.

It also compounds across runs. The second drive reads the same queue, applies the same reasoning, reaches for the same attractive item, and passes over the same four — so the skipped set is stable rather than random, and the items least likely to be done are precisely the ones that have already been passed over several times.

Why the obvious fix failed#

Order the queue properly. Necessary and not sufficient. A correctly ordered queue is still a list a drive can read all of, and ordering expresses a preference rather than a constraint — nothing stops the second item being started first.

Mark items as blocking. Helps for the dependencies you know about. Most skips are of items whose prerequisite relationship nobody wrote down, usually because it was obvious to the person who created them.

Tell the drive to work in order. An instruction competing with an objective, and the objective is more specific and more rewarded. When the two conflict, the instruction loses quietly.

Review the queue afterwards. Detection rather than prevention, and it happens after the run has reported success. By then the skipped items are indistinguishable from items that were never going to be done this time.

The actual mechanism#

Lock the queue so that only the head is available, and require an explicit disposition before it moves.

head      only the front item is visible to the drive
dispatch  do the work, then advance
dequeue   explicitly drop it, with a reason, then advance
never     reach past the head to anything behind it

The mechanism is not really about ordering — it is about making the skip into an action. Dequeuing with a reason is permitted and cheap; what is removed is the option of passing over an item without saying anything, which was the only thing wrong with the original behaviour.

That reframing is what makes the rule tolerable in an autonomous drive. A head lock that forbade dropping items would be a straitjacket, and a drive that must occasionally decide an item is noise needs to be able to say so. The constraint is on silence, not on judgement.

The sub-case that makes it work in practice is what happens when the head cannot be done. A prerequisite is missing, a device is unavailable, an input has not arrived. The answer cannot be "move on", because that is the original behaviour; it has to be either a halt or a dequeue with a recorded reason — both of which are visible, and either of which is better than a silent pass.

The halt-versus-dequeue choice is worth deciding deliberately rather than per-case. A missing prerequisite that will arrive is a halt; one that will never arrive is a dequeue; and a drive that treats everything as a dequeue converges on an empty queue and no work done, which is the failure this mechanism can produce if it is applied without the distinction.

There is a second sub-case that is easy to miss: items added during the run. Work frequently generates more work, and if new items go to the front, a drive can live at the head forever without the original backlog moving at all. New items go behind, and the exception — a genuine prerequisite discovered mid-item — is itself a dispositioned decision rather than a default.

Discovered work going to the back feels wrong the first time, because the new item is often more urgent than the head. It is still the right default: urgency is exactly the argument that produced the original skipping, and permitting it as a default reintroduces the behaviour through a door marked prioritisation.

The fix#

lock      expose only the head item to the drive
dispose   each item ends as dispatched or dequeued-with-reason
append    newly discovered work goes to the back, not the front
audit     the run reports every disposition, including the dequeues

The lock is a type, not a discipline — the drive is never handed the collection:

type Disposition =
  | { kind: "dispatched"; result: unknown }
  | { kind: "dequeued"; reason: string }        // permitted, cheap, and RECORDED
  | { kind: "halt"; reason: string };           // the prerequisite will arrive; stop

/**
 * Only `head()` is exposed. There is deliberately no `items`, no `find`, no index access —
 * the skip is not forbidden by a rule the drive could forget, it is absent from the surface.
 */
class HeadLockedQueue<T> {
  #items: T[];
  #audit: Array<{ item: T; disposition: Disposition }> = [];

  constructor(items: T[]) { this.#items = [...items]; }

  head(): T | undefined { return this.#items[0]; }

  /** The only way the head moves. A disposition is required to advance. */
  advance(d: Disposition): void {
    const item = this.#items.shift();
    if (item === undefined) throw new Error("advance() on an empty queue");
    this.#audit.push({ item, disposition: d });
  }

  /** Discovered work goes to the BACK. Urgency is the argument that produced the skipping. */
  append(...items: T[]): void { this.#items.push(...items); }

  report(): string {
    const by = new Map<string, number>();
    for (const a of this.#audit) by.set(a.disposition.kind, (by.get(a.disposition.kind) ?? 0) + 1);
    const dequeues = this.#audit
      .filter((a) => a.disposition.kind === "dequeued")
      .map((a) => `  dequeued: ${String(a.item)} — ${(a.disposition as { reason: string }).reason}`);
    return [`dispositions: ${[...by].map(([k, n]) => `${k}=${n}`).join(" ")}`, ...dequeues].join("\n");
  }
}

async function drive<T>(q: HeadLockedQueue<T>): Promise<void> {
  for (let item = q.head(); item !== undefined; item = q.head()) {
    const d = await attempt(item);          // returns a Disposition, never void
    if (d.kind === "halt") { console.log(q.report()); return; }
    q.advance(d);
  }
  console.log(q.report());
}

And the output, which is the half that makes the drive auditable from outside:

$ deno task drive --queue precondition_dispatch_queue

dispositions: dispatched=14 dequeued=4
  dequeued: regenerate-flows(feature=onboarding) — superseded by the enrich in item 2
  dequeued: device-test(scanner) — no device attached and none will be in this run
  dequeued: locale-sweep(ga-IE) — locale removed from the project this morning
  dequeued: retrain-corpus — ran in the previous drive 40 minutes ago, nothing changed since

Four dequeues with reasons is a claim a reader can disagree with — and the fourth one is wrong, which is the point. Something had changed since that previous drive, and the reason being written down is what let somebody notice. The same eighteen items before the head lock reported dispatched=14 and nothing else, and the four it passed over were indistinguishable from items that had never been queued.

Note that attempt returns a Disposition rather than void or a boolean. That signature is doing most of the work: a function returning void can decline to act and say nothing, which is the original behaviour, and there is no amount of care at the call site that recovers the reason it declined.

The fourth line is what turns the mechanism into something you can trust from outside. A drive that dequeued six items with reasons is making a claim that can be read and disagreed with; the same drive without the audit is indistinguishable from one that did the work.

The dequeue reasons are also, in my experience, the most informative part of a run's output. They are the places where the drive encountered something it could not or should not do, written down at the moment it happened — which is a much better record than anything reconstructed afterwards.

What this costs is some throughput. A drive that must take items in order cannot batch opportunistically or skip to the thing it is best placed to do right now, and on a queue where order genuinely does not matter, that is pure overhead. The trade is worth it because the queues where order does not matter are not the ones that cause incidents.

There is also a quieter benefit that partly offsets the cost. A drive working one item at a time produces a legible run — each disposition in sequence, each with its own outcome — where an opportunistic drive produces a narrative that has to be reconstructed to be understood.

How we proved it#

The skip is reproducible and the failure is not. Give a drive a queue with a setup item and an attractive goal item, and watch which one it starts. The skip happens reliably; what it breaks depends entirely on what the setup item was for, which is why the symptom was so hard to recognise as a pattern.

The dequeue reasons surfaced real gaps. Once skipping required a stated reason, several items turned out to be dequeued with variations of "no longer applicable" — which was true, and meant the queue had been accumulating stale work that nobody had noticed because it was always being passed over.

What I have not measured is the throughput cost. The head lock is obviously slower in principle and I have not instrumented how much, partly because the comparison is awkward: the unconstrained version sometimes fails later, and a fair measurement would have to price that in.

What I have not solved: genuinely parallel work. A head lock serialises, and some queues contain items with no relationship to each other at all. The right answer is presumably per-lane locks rather than one global one, and I have not built that — so a drive with independent work does it one at a time.

And a limitation worth naming: the lock enforces disposition, not quality. A drive can dequeue an item with a plausible reason that is wrong, and the audit will record a well-formed decision. This converts a silent skip into a stated one, which is a real improvement and is not the same as making the decision correct.

The thing that reframed it was noticing that the drive was not disobeying anything — it was prioritising, which is what you want it to do, against a goal that did not mention the queue. The behaviour follows from the objective, so it cannot be fixed by an instruction that competes with the objective.

Which suggests the general shape of the remedy: when a behaviour follows from an objective, change what is available rather than what is asked for. A drive that cannot see past the head does not need to be told not to skip.

What this generalizes to#

An autonomous process optimising for a goal will treat anything not required by the goal as optional, including the things that were going to be required later.

The same dynamic shows up wherever a worker has visibility over a whole backlog: a build that skips to the target it was asked for, a migration runner that applies the one you named, a person working a ticket list by what looks tractable. In each case the reordering is rational and the dependencies nobody declared are the casualty.

So the question to ask of any backlog-consuming process is what stops it from choosing. If the answer is an instruction, it will choose anyway — and the cheapest fix is almost always to show it less rather than to tell it more.