The 504 That Was Actually Working

I had just put a side project on the public internet for the first time — a European job search, a Laravel and Inertia app with a Vue front end, built on a snapshot of a public board — and I had deployed it to Fly.io the way I like best, with almost no moving parts. The entire database is a single read-only SQLite file, 1.1 gigabytes of it, baked directly into the deploy image and shipped inside the container next to the code. There is no database server to run and no managed instance to point at; refreshing the data is just another deploy. For a read-only app it is a wonderfully boring setup, which is the highest compliment I give infrastructure. The build went green, the machine came up healthy on the dashboard, and I opened the URL with that small, specific pleasure of watching something you made resolve in a browser for the first time.

What resolved was a 504 Gateway Time-out, in the flat grey of an error page that has stopped trying.

A timeout with a healthy machine

My first move was denial, which is everyone's first move: I reloaded, and got the same 504 exactly sixty seconds later, to the second. I opened the Fly.io dashboard and the machine was sitting right there — awake, healthy, no crash loop, no out-of-memory kill, nothing in the logs with the shape of a stack trace. Every signal the platform offered me said the application was fine. It simply would not answer the door.

HTTP/2 504
server: nginx

So I SSH'd onto the running machine to interrogate it directly, and the picture got stranger rather than clearer. The database file was exactly where it should be, all 1.1 gigabytes present and accounted for. The health endpoint — the little /up route Laravel ships for precisely this kind of moment — came back instantly. The page I actually wanted did not.

GET /up   → 200   in 0.29s
GET /     → 504   at 60.00s

Those two lines are the whole mystery. The same machine could answer one route in under a third of a second and could not answer the other before nginx gave up at sixty. The lobby light was plainly on while the front door timed out. Whatever was wrong was not the machine; it was something specific to the one page I cared about.

The database had an alibi

My first real suspect was the database, and I want to be honest that it was a good suspect. SQLite's query planner leans heavily on a set of statistics that a command called ANALYZE produces, and when those statistics are missing or stale the planner will cheerfully choose the wrong index and turn a sub-hundred-millisecond query into one that takes five, ten, thirty seconds. This very project had been bitten by exactly that before. A cold machine, freshly deployed, serving a query that hangs for half a minute is precisely the silhouette that ANALYZE trouble casts — so reaching for the database was not a wild stab, it was the single most plausible explanation on the board.

I checked it the only way worth checking, by looking at whether the statistics were actually there:

sqlite> SELECT count(*) FROM sqlite_stat1;
36

Thirty-six rows of statistics, which meant ANALYZE had run during the import and the planner had everything it needed. The database had an alibi. While I was in the mood, I ruled out the other familiar ghost — server-side rendering, which runs in its own little Node process and can hang in genuinely interesting ways — by poking its port and watching it refuse the connection instantly, which is the difference between "not running" and "hanging." Two of my three best theories were dead inside ten minutes, the page was still timing out, and I was no closer to why.

183 seconds

The obvious thing I had somehow not yet done was time the page — not through nginx, which could only ever tell me "longer than sixty seconds," but underneath it, asking the application itself how long it took to assemble the response. So I dropped into Laravel's tinker shell and rendered the page by hand, twice.

1st render:  183.2s   → 200 OK
2nd render:   21.1s   → 200 OK

There it was, and it rearranged the entire problem. The page was not broken. It was returning a flawless 200 — it was just taking a hundred and eighty-three seconds to produce it the first time, and twenty-one on every visit after. Nothing had crashed. The page was thinking.

What it was thinking about was relevance. Every search runs its matching listings through a composite scorer that pulls each row into PHP and computes a score for it, one at a time, before ranking the set — and on the broadest possible query, the one a visitor sees before they have narrowed anything, that is the entire corpus, something like thirteen thousand listings, scored in a PHP loop. Twenty-one seconds of honest arithmetic, warm. The first render is so much worse because everything is cold at once: the PHP opcode cache is empty, the app's own caches are empty, and that 1.1-gigabyte SQLite file has not yet been pulled into the operating system's page cache, so even the disk reads are paying a one-time warmup tax. SQLite was in the room — just as a contributor to the cold tax, never as the bill itself.

And then the decision that turned a slow page into a broken one. To keep a hobby project cheap, I had told the Fly.io machine to scale to zero whenever it went idle. It is a lovely idea right up until you notice what it means: every visit after a quiet spell now wakes a cold machine and pays the full hundred-and-eighty-three-second tax, and nginx, not unreasonably, stops waiting at sixty. The 504 was not an error report at all. It was a stopwatch.

The cache that can't be wrong

The fix came in three parts, and only one of them is interesting. The two dull ones first: I stopped the Fly.io machine from scaling to zero, so a single instance stays warm with its opcode cache and its page cache intact, and I raised the timeouts — nginx's and PHP's both — to five minutes, so that on a genuinely cold render the page is allowed to finish rather than being cut off at sixty seconds. Together those turn the worst case from "broken" into "slow exactly once."

The interesting part is the cache. I wrapped the whole scored page in Laravel's Cache::remember, keyed on the exact filter state, so the expensive thirteen-thousand-row loop runs once per distinct view and every visit afterward is served from cache in milliseconds:

$props = app()->isProduction()
    ? Cache::remember(\(key, now()->addDay(), fn () => \)this->buildProps(...))
    : $this->buildProps(...);

And here the deployment model I picked at the very beginning quietly pays for itself. Caching is usually the easy half of a hard problem; the hard half is invalidation — knowing the precise moment a cached answer has gone stale and tearing it down. I wrote none of that, and I never will, because the data this cache is built from cannot change underneath it. The corpus is a read-only SQLite file baked into the image, and the only way the data ever changes is a new deploy — which ships a new image, a fresh machine, a fresh and empty cache. The file and the cache are replaced together, atomically, by the single act of deploying.

A cached page here isn't "probably still fresh." It is structurally incapable of being stale — because the only thing that could spoil it also throws it away.

Had a managed Postgres sat behind this app, with the application writing to it, that paragraph would be a lie, and I would owe the codebase a careful, bug-prone invalidation layer for the rest of its life. The read-only-baked-SQLite model is exactly what lets me delete that entire category of work. The database that looked so guilty in the second act turns out to be the reason the fix is clean.

What the error was telling me

I think this one cost me an extra beat because 504 reads as a crash. It lives in the same mental drawer as 500 and segfaults and the stack traces you go hunting through — the drawer labelled something is broken. But nothing here was broken. The machine was healthy, the file was intact, the query was correct, and the page returned a perfect 200 the moment you gave it enough room. The error was never a failure; it was a mismatch — between how long the page took to think and how long the proxy in front of it had agreed to wait. There was no bug in the ordinary sense to find, because the bug was a decision: I had parked a cold-expensive page behind a scale-to-zero switch, and the timeout was simply the seam where those two choices met.

The lesson I am keeping is a small and slightly embarrassing one about reading errors literally. A 504 does not mean broken. It means this took too long — and "too long" is a question with two halves, how long the work takes and how long you have agreed to wait for it, and the fix can live in either half. I spent my first hour certain the work was wrong. The work was fine. I had simply never told anyone how long it was allowed to take.

---

Related: The Job Finder That Reads Every Listing — the project this gateway timeout came from, an end-to-end tour of the AI pipeline that reads every listing and the app it ships in. And The Endpoint That Moved While I Wasn't Looking — another deploy-day investigation where the obvious, loudly-announced explanation turned out to be the wrong one. And The Secret Your Container Has but Your Shell Can't See, on a database password that quietly vanished inside docker exec.