Never attempted, and the summary said complete

A list of items is read line by line and each one is processed. The run finishes, reports what it did, and declares itself complete.

The last line of the file was never read. The file did not end with a newline, and the loop treats that as nothing remaining.

The case this post is about#

One file of forty-nine items, and a loop that attempted forty-eight.

The last line had no trailing newline. An ordinary read loop reads up to a delimiter and reports failure when it does not find one — so the final item is read into the variable and the loop exits before the body runs on it.

Nothing errored. The loop completed, the summary said done, and one item was never attempted. It was not skipped with a reason or recorded as failed; it simply did not happen, and the only evidence is a count nobody was keeping.

This post follows that one missing line, because the loop is correct shell, the file is a valid file, and the fix is an independent count rather than a better loop.

The symptom#

An item that was never attempted. Not failed, not skipped with a reason, not retried and abandoned — absent from the entire run, as though it had never been in the list. There is no error to search for, because nothing went wrong from the program's point of view.

The second thing is that the report is internally consistent, which is what makes it convincing. The summary counts what was processed and says all of it succeeded, and both halves are true. A report is a statement about what happened; it has no independent knowledge of what should have happened, so it cannot contradict itself about a thing it never saw.

The third is why the file lacked a newline, which is never a decision. It was written by an editor configured not to add one, or produced by a command whose output does not end in one, or truncated by a copy. Nothing in the content indicates it, and the file looks identical in every viewer that does not go out of its way to show you.

That invisibility is worth dwelling on, because it rules out the normal response to a data problem. You cannot look at the file and see it. You cannot diff two files and see it unless the diff is configured to say so. The distinguishing byte is the absence of a byte, and almost nothing in a normal working day is capable of showing you an absence at the end of a file.

The fourth is the position of the loss, which is the cruel part. It is always the last item. In a list that grows by appending, the last item is the newest — so the thing that goes missing is reliably the most recently added, which is the one most likely to be the reason the run was happening at all.

That is not a coincidence, it is the mechanism. Appending puts the newest item last and the loop drops the last item, so the two line up perfectly every time — which means the failure is at its most likely precisely in the run where somebody has just added something and is watching to see it go through.

Why the obvious fix failed#

Check the exit code. Zero. Everything the loop attempted succeeded, and the loop attempted everything it read. Exit codes report on execution, and this is a failure of enumeration, which happens before any execution exists to report on.

Read the summary. The summary is produced from the same enumeration as the work. It says "processed 48, succeeded 48" and both numbers are correct and both are wrong, because the denominator was computed from the same truncated list.

Add error handling to the loop body. The body never ran for the missing item. Error handling is for things that go wrong during processing, and nothing went wrong during processing — there was no processing.

Spot it in review. A list of forty-nine items and a report of forty-eight is a discrepancy nobody notices, because the report does not state the expected number and nobody counts the file by hand before running something.

The absence of an expected number is what makes the whole thing invisible, and it is a reporting choice rather than an oversight in the loop. Somebody decided what the summary should say, and "processed 48" seemed complete because it names a quantity. A quantity without its denominator reads as precision and carries none.

The actual mechanism#

Count the input independently of the loop that consumes it, and assert the two agree.

count     the items before processing, by a different means than the loop
process   the list
compare   attempted against counted, and fail loudly if they differ
read      the file in a way that does not depend on a trailing delimiter

The third line is the generalisable part and it costs two lines of code. An expected count obtained before the work begins, compared against the attempted count afterwards, converts a silent omission into an immediate failure — and it catches every variant of this, not just the newline one.

The fourth line is the specific fix and it is worth knowing because the pattern is so common. A read loop that waits for a delimiter treats a final unterminated fragment as nothing. Reading the whole input and splitting it, or explicitly handling a non-empty remainder after the final delimiter, both avoid it; so does ensuring the file always ends with one, which is a convention rather than a guarantee.

Relying on the convention is reasonable within a repository you control and stops being reasonable the moment a file arrives from anywhere else — a download, an export, another team, a tool's output. The convention is about files you write; the loop has to survive files you receive.

The sub-case worth naming is where the count comes from. If the expected number is derived from the same read that feeds the loop, the comparison is a tautology and will agree forever. The two counts have to be obtained by genuinely different routes — one from a line count, one from iterations performed — or the check validates nothing while appearing rigorous.

This is the failure mode of a great many well-intentioned assertions. Two values that came from the same computation will always agree, so the check passes forever, and its passing is taken as evidence about the world rather than as a statement that a number equals itself.

The fix#

count      expected, independently, before starting
assert     attempted equals expected, after finishing
read       in a way that handles an unterminated final item
state      both numbers in the summary, so a reader can see the comparison

The independent count is the mechanism — counting while you iterate uses the same reader that drops the line, so it agrees with itself and with nothing else:

process_all() {
  local file="$1"

  # INDEPENDENT count, by a different reader. `grep -c` counts matching lines
  # including an unterminated final one; the loop below may not.
  expected=$(grep -c '' "$file")

  attempted=0
  # `|| [ -n "$line" ]` is the fix for the dropped line: a final item with no
  # trailing newline leaves `read` returning non-zero WITH the data in $line.
  while IFS= read -r line || [ -n "$line" ]; do
    [ -z "$line" ] && continue
    attempted=$(( attempted + 1 ))
    handle "$line"
  done < "$file"

  if [ "$attempted" -ne "$expected" ]; then
    echo "DROPPED: attempted $attempted of $expected from $file" >&2
    return 4
  fi
  echo "processed $attempted/$expected"
}

A file whose last line has no trailing newline — which is what a generator, or any write that does not go through an editor, produces:

$ process_all slugs.txt          # without the guard
processed 157

$ process_all slugs.txt          # with it
DROPPED: attempted 157 of 158 from slugs.txt

The first output is not wrong about anything it says. It simply has no opinion about the line it never saw, and 157 looks exactly as complete as 158 when 158 was never printed.

The fourth line is the cheap insurance for every case the assertion does not cover. A summary saying "48 of 49" is self-evidently wrong to anybody glancing at it; a summary saying "48 processed" is self-evidently fine, and the difference between those two outputs is one number that the program already has.

What makes this worth generalising rather than patching is that the newline is incidental. The structural problem is a loop that defines its own completeness — it processes what it finds and reports on what it processed, which is a closed statement that cannot detect its own incompleteness no matter what caused it.

How we proved it#

The missing item is identifiable by comparison in seconds. Count the lines in the input, count the entries in the output, and the difference names what was not attempted. The whole difficulty was that nothing in the normal path put those two numbers next to each other.

The newline is confirmable directly. Looking at the final byte of the file settles the cause immediately — and it is worth doing rather than assuming, because an off-by-one in the loop produces identical symptoms and has a different fix entirely.

What I cannot tell you is how many earlier runs lost their last item. The reports from previous runs are all internally consistent and none of them records an expected count, so there is nothing to re-examine. Every past run that ended on an unterminated file would have looked exactly like a success.

What I have not solved: the same class in places where the enumeration is not a file. A list from a paginated API, a directory listing bounded by a limit, a query with an implicit row cap — all have the same property of defining their own completeness, and the independent-count trick needs a genuinely independent source, which is not always available.

Where it genuinely is not, the next best thing is to make the boundary explicit rather than pretending it is not there — recording that the enumeration came from a source with no independent total, so a later reader knows the completeness claim is conditional. That is weaker than a check and much stronger than silence.

And a limitation of the assertion: it tells you the numbers disagree and not why. That is enough to stop a run and almost never enough to fix it, so the value is entirely in failing rather than in diagnosing. I have come to think that is the correct division — a check that stops the wrong thing happening does not also need to explain it.

The thing that reframed it was noticing that the loop and the report share a single source of truth, which makes the report incapable of auditing the loop. Every self-reported summary has this property to some degree, and the only way out is a number that was obtained before the work and does not depend on it.

That number is usually trivial to obtain and almost never obtained, which is the lesson in miniature. Nobody skips it because it is hard; they skip it because the loop already produced a count, and having a count makes the absence of a second one invisible.

What this generalizes to#

A process that reports on itself cannot report what it never saw, and most completeness claims are exactly that kind of report.

The pattern is wherever a producer and its own audit share an enumeration. A backup that reports every file it copied, from a walk that skipped a permission-denied directory. A migration that reports every row it transformed, from a query with a default limit. A deployment that reports every service it updated, from a manifest that was missing one. In each case the work and the proof come from the same list, so the list's omissions are invisible to both.

So the question worth asking of any "all done" is where did the denominator come from — because if the answer is "the same place as the numerator", the claim is that the process did what it did, which is true of every process that has ever run.

The useful follow-up is to ask what the denominator would have to come from to be independent, and to notice when the honest answer is that nothing available is. That happens — a paginated API with no total is genuinely unauditable from the client — and knowing it is far better than assuming the numbers mean what they appear to.