Debug: The Dropped Page

This one starts broken. You are handed working code that reviews cleanly, runs without an error, and loses the last page of every sync. Read it before you type.

The diagnosis

Here is the shipped bug, close to verbatim:

let page = await fetchPage(null);

while (page.nextCursor) {
  for (const record of page.records) {
    await onRecord(record);
  }
  page = await fetchPage(page.nextCursor);
}

Walk it with two pages. Page 1 comes back with nextCursor: "c2", so the loop body runs: its records are delivered, and page 2 is fetched. Now the condition is tested again. Page 2's nextCursor is null, the loop exits, and page 2 is sitting in page with every one of its records undelivered. The bug fetched the data. It just never used it.

That is the shape to learn, because it is not really about cursors:

Say before you type: "The loop condition is being asked about a page I have not drained yet."

A while condition guards entry, and this one is testing a property of the page the body is about to skip. Any loop that fetches at the bottom and tests at the top is one page behind itself. With one page it happens to be right, which is why the unit test suite was green.

The fix, and the patch that sets a trap

Move the terminal check below the drain:

let cursor: string | null = null;

while (true) {
  const page = await fetchPage(cursor);

  for (const record of page.records) {
    await onRecord(record);
  }

  if (page.nextCursor == null) return;
  cursor = page.nextCursor;
}

Drain the page you are holding. Then ask whether another one exists. That order is the whole lesson.

The tempting alternative is to leave the loop condition alone and drain the leftover page after it:

} // end while
for (const record of page.records) await onRecord(record); // the leftover drain

Be honest about this one, because half the value of a bug squash is not overclaiming. Bolted onto the untouched starter, that line does deliver every record exactly once, and the bench accepts it. It is still the worse fix. The delivery code now lives in two places, the loop condition is still being asked about a page nobody has drained, and the leftover drain is correct only because the loop happens to stop one page short. Change the loop to a do/while, or to the drain-then-check shape above, and that same line delivers the last page twice.

Two writes is not better than zero writes. It is a different incident with a worse cleanup. The bench counts deliveries per id for exactly this reason, and so does your database.

While you are in there: fetch each page once. "Re-fetch to be safe" is how a page turns into two pages of duplicate work.

Replay safety

Real syncs run again. Take seen, a Set of ids already delivered, skip anything in it, and return { delivered, skipped }.

The whole phase is one line's placement:

await onRecord(record);
seen.add(record.id);   // AFTER, always after

Mark the id first and a throwing handler leaves you with a record that is recorded as done and was never delivered. The retry then skips the single record it existed to deliver. Marking after means a crash costs you at most one redelivery, and redelivery is what seen is for.

Add ids to the same seen you were handed, not to a snapshot of it. Then an id that shows up on two pages inside one run is delivered once and skipped once, for free.

Resumable checkpoints

options.startCursor says where to begin. options.onCheckpoint(cursor) says how far you got.

Call onCheckpoint(page.nextCursor) once per page, after the last record on that page has been delivered or skipped. Not per record. nextCursor names a page boundary, and a boundary is only true once the page behind it is finished. Checkpoint after each record and you save a cursor that points past a page you were halfway through, which means the saved position claims more progress than actually happened.

Checkpoint the last page too. Its cursor is null, and null is the signal that the walk finished rather than died.

Then the crash story works: the handler throws on page 2, the exception propagates, and the last checkpoint still reads "c2". Resume with that cursor and the same seen, and page 2 is re-fetched, its delivered records are skipped, and the rest of the job finishes. Exactly once, end to end, across a process that died in the middle.

What an interviewer is listening for

Name the invariant, not the patch. "Every record reaches the handler exactly once" is the property. The dropped page and the double-delivered page are the two ways to break it, and a fix that trades one for the other has not moved you.