My Overnight Chain Killed a Job That Had Already Finished

This morning I woke up, opened Slack, and the one message I actually depend on wasn't there.

Every night my homelab runs a chain of more than two dozen steps. It reads the news, reviews my repos, audits itself, and then, at the very end, it writes me a single briefing: what happened overnight, what broke, what needs my hands today. That briefing is the entire point. The other steps are just reporters. This morning the reporters filed and the paper never printed.

A briefing is only as fast as the slowest thing it waits on

The morning-briefing step doesn't run first. It runs last, because it summarizes everything before it — so it sits and waits on fourteen upstream steps to finish. One of those is article-scanner, the worker that pulls north of a hundred and fifty links a night and decides which handful I'd actually want to see.

Here's the number I wasn't watching: article-scanner used to take about six minutes. This morning it took forty-eight.

The timer gave up at forty-five. The worker finished at forty-nine.

My chain has a safety valve. If any single step runs longer than forty-five minutes, a reaper assumes it's wedged, marks it failed, and moves on — otherwise one hung worker could hold the whole night hostage. It's a good rule. It has saved me before.

So at minute forty-five, the reaper looked at article-scanner, saw it still going, and killed the branch. That one "failure" cascaded: twenty downstream steps got skipped because they depended on it, and the last one in that line was my briefing.

Then the part that took me a second read to believe. At minute forty-nine — four minutes after the reaper had already written it off — article-scanner finished. Cleanly. It posted its digest to Slack like nothing was wrong. The worker didn't fail. It won the race. My own clock just wasn't willing to wait for the photo finish.

Why a six-minute job became a forty-eight-minute one

Nothing broke. The scanner got heavier, one quiet feature at a time. Over a few weeks I'd taught it to do two more things after it picks the links: deep-read the best articles into my memory layer, and draft code-change proposals from the top ones. Useful work. Slow work. All of it running inline, in series, before the digest ever went out.

So the runtime crept — six minutes, then twenty, then forty, then forty-eight — and I never noticed, because the briefing kept arriving anyway. Right up until the night it didn't. This is how automation actually fails: not with a crash, but with a number that drifts for a month while every morning looks fine.

The fix I shipped first was a lie I told myself

Honest confession: my first move was to raise the timeout from forty-five minutes to seventy-five. It worked. Tomorrow's briefing was safe. And it was a band-aid, and I knew it while I was typing it — I had moved the cliff, not removed it. Give that scanner another few weeks of quiet feature creep and I'm right back here, writing this same post with bigger numbers.

I shipped it anyway, because a working briefing tomorrow beats a perfect briefing next week. Then I went back for the real one.

The briefing was waiting on work it never reads

Here's the thing I should have seen immediately. The two slow jobs article-scanner had learned to do — the deep reads, the proposals — the briefing doesn't use either of them. They feed my memory layer and a separate code-fixing worker. The briefing only ever needed the short list of links. It had been sitting in the dark for forty minutes waiting on work that was never addressed to it.

So I split the worker in two. It now does the fast part — fetch, score, hand me the list — and stops. The heavy forty-minute half moved into its own step that runs off to the side, where nothing waits on it. The briefing now depends on a ten-minute job instead of a fifty-minute one.

I proved it on a live run before I trusted it: the fast scan finished in under two minutes, the heavy half ran on its own afterward, the briefing fired on time, and nothing downstream got skipped. Then I deleted the seventy-five-minute band-aid I'd left as a net.

Keep the critical path short

The lesson isn't about timeouts. It's that the thing you depend on every single morning should never be gated by the slowest, least-important work of the night. My briefing had quietly become hostage to a background chore, and the only reason I ever found out was that the chore finally crossed a line I'd drawn months ago and forgotten.

The fix is almost never a bigger timeout. It's asking what the important thing is actually waiting for, and then cutting it loose from everything that isn't the point.

My reporters can take all night if they want. The paper still has to print by morning.