This time I had been sure. Twelve testers had stayed in my closed test for the whole window, most of them opening the app every day, and I had shipped two releases five days apart, all of it tracked in my own database. None of that moved the answer.

It took four applications and nearly five months to get Akluma onto Google Play, and each rejection corrected a wrong idea I held about what the review measures. The application that finally passed came from a round in which fewer people opened the app each day than in the round Google had just turned down.

Application 1   11 May        rejected 13 May
Application 2   17 July       rejected 25 July
Application 3   18 August     rejected 21 August
Application 4   10 September  approved 12 September

Akluma is a free savings tracker I built with Laravel: you set a savings goal, it builds a dated plan and emails you a reminder on each saving date, and it recalculates the plan when life gets in the way. I wrote about how that plan rewrites itself in an earlier post. This one is about getting it into a store.

A web app in an Android shell

Akluma is a web app, and why it needed a store listing at all is a story I told in a post about my keyword research: most people find apps by searching inside the store, and my research had been blind to that whole channel. The way to put a web app into Google Play is a Trusted Web Activity, a thin Android app whose only job is to open your website full screen in Chrome. There is no second codebase. The app on your phone is akluma.com without the address bar, running against the same server and the same database, and every web deploy reaches it the moment it ships.

Setting that up took about a week in April, and only one detail from it is worth passing on. The file that proves you own the domain, assetlinks.json, has to carry the fingerprint of Google's app signing key and not only the key you upload with, because Google re-signs your app before it reaches anyone's phone. Leave Google's fingerprint out and the address bar appears inside your app.

Then came the requirement I had not planned for. Google asks new personal developer accounts to run a closed test with at least 12 testers opted in for 14 continuous days before they can even apply for production. My identity was verified on 20 April, the first release went to closed testing on 24 April, and twelve testers for fourteen days sounded like arithmetic.

Round one: twelve people and a calendar

My first theory was exactly that: the gate is arithmetic. Find twelve people, keep them enrolled for fourteen days, apply. So I asked family and friends over WhatsApp to install the app and, above all, not to tap "Leave the program". I did not ask anyone to use it, because by the letter of the rule nobody had to.

I even got the arithmetic wrong at first, holding back my submission because I had read that the 14-day clock restarts with every tester you add. It starts when testers opt in, and the opt-in link only exists once Google has published your release, so you have to submit before you can recruit anyone. The clock has stranger habits than that, which get their own post later in this series.

I applied for production on 11 May, and two days later the first rejection arrived. Its first possible reason was "Testers were not engaged with your app during your closed test."

It was right. When I finally audited that round in July, only 3 of the 13 people on my list had ever registered an account in the app, and one of the three was me.

One good thing did come out of it. On 14 May my wife, who had been using Akluma on the web for most of a year, left this in the closed test's feedback:

It's a very useful app to set and track saving goals. Easy to use and convenient with the automatic email reminders. What I would suggest adding is the possibility to choose or change the date for the automatic reminder (for example so you can set it around the time when your salary is paid or similar).

I filed it as a good idea for later. It would matter far more than I expected.

Round two: paying for engagement

By June I understood that engagement was the problem, and my second theory was that engagement could be bought. Swapping tests with other developers cost more hours than I had, so I paid a Fiverr seller $30 for a batch of testers. When I checked my database on 9 July, 12 of the 31 people on the list had registered, 4 had an active savings goal, and exactly one had ever moved money. The seller offered me a ready-made answer sheet for Google's application form that described improvements I had never made, and I did not use it. Around 17 July the testers' five-star reviews vanished from my console, most likely removed by Google's checks for coordinated reviews, though that is my inference. Fiverr refunded the order at the end of July, and through all of August, Play's statistics still counted around ten of those testers with the app installed and not one of them opening it.

The real lesson of that round came from a question I should have asked in April: what can Google actually see? Akluma is a Trusted Web Activity, which means that everything a tester does inside it, from registering to creating a goal to adding money, happens in Chrome. Play sees none of it. What Play can see is whether the app gets opened, and on how many different days. I had spent a month asking people to create piggy banks, and piggy banks are invisible to Google.

I applied again on 17 July, with answers I wrote myself and kept honest. On 25 July the same email came back.

Round three: measuring what Google measures

My third theory was that if Play counts opens, I should count them too, and drive them on purpose. Before asking anyone for anything, I built the instrument: one row per user per day, the installed app told apart from a browser tab by the referrer Android attaches at launch, a heartbeat counting seconds on screen, and an admin page showing exactly which testers had gone quiet. It went live on 3 and 4 August, and I will come back to how it works.

Then I designed the ask around the one habit Akluma naturally creates. A savings plan can be daily, weekly or monthly, and every saving date sends a reminder email at nine in the morning. So I asked each tester to create one small daily piggy bank, a couple of weeks toward something modest. A daily plan produces about thirteen reasons to open the app in fourteen days, where a weekly one produces two.

From 5 August I messaged testers one by one on WhatsApp almost every day, and 12 of the 13 engaged, nearly all of them daily.

I also shipped. Version 4 went out on 13 August with the feature my wife had asked for in May: on a monthly plan, you can now choose the day of the month your reminders arrive. Version 5 followed on 18 August and replaced a dashboard that had been showing placeholder figures. Those releases mattered twice over, because a web deploy is invisible to Play, and the only update Google can see is a new bundle on the testing track.

I applied on 18 August. On 21 August I got the email at the top of this post.

What I told Google, and what it could see

Play Console has a page called Testing feedback, under Ratings and reviews. It is where testers' private feedback lands when they use the "Private feedback to developer" control on the app's Play Store page, and I had looked at it plenty of times over the summer. I never treated it as evidence. The application form asks how you collected feedback from your testers, and I had answered truthfully: over WhatsApp. Once I had told Google where the feedback lived, I assumed it made no difference whether any of it also appeared on that page. It did not help that after the paid reviews vanished, I had stopped asking testers to write anything on Play at all.

The day after the third rejection, I read the page the way a reviewer would. Filtered to all time, it held one item: my wife's, from 14 May, on version code 1. Across the whole span Google had just evaluated there was no new feedback, and the only activity was my own reply to her, posted on 13 August, telling her that the feature she had asked for was live. Meanwhile that round had produced three useful new pieces of feedback, and all three sat in WhatsApp, where Google has no way to look.

Google's requirements page for new accounts asks you to "summarize your testing feedback when applying for production access", and I had. What I had not given Google was any way to check the summary. From the reviewer's side of the table, my answer about feedback was a claim with nothing behind it.

I want to be careful here, because Google never told me this was the reason. It gave no reason at all, three times, and a support ticket asking what exactly had been judged insufficient came back with a list of best practices. That near-empty page is one of three explanations I believe fit. The second is that my testers' usage looked instructed, because it was. The third is that numbers which looked healthy over my own fourteen days looked thin over Google's twenty-four, which began at the previous review date on 25 July, eleven days before I began onboarding my testers.

Two other theories died over the following weeks. Google's support told me to fix the issues in my pre-launch report, which had never been generated, and after six uploads support admitted that it cannot generate below an activity threshold they would not name. And for one day I was convinced Play could not see a Trusted Web Activity at all, until Play's own chart of daily users showed that it could.

Round four: evidence where Google can see it

The fourth round started without me. Google's 14-day counter restarts on its own at each review date, and my twelve testers had never left the programme, so the app became eligible to apply again around 5 September without a single new recruit. That left me with one simple rule for tester lists: add people, never swap them, because a single dip below twelve restarts the count.

I did try to add people. On 24 and 25 August I emailed 32 existing users of the web app, a personal note to each, inviting them to join the test. None of them did.

So the round came down to the one thing I could still change: putting evidence where Google looks. On 25 August I asked my family and friends to leave private feedback on the app's Play page, and instead of a general request I gave them two questions, in Turkish, that amount to: what would you want to be different in the app, and which part confused you? A general ask produces "great app". A specific question produces something you can act on.

Eight pieces of feedback arrived over the next four days, and versions 7 and 8 shipped within two days of them, answering four. The buttons for adding money moved to the top of a goal's page, and every tappable row on the dashboard gained an arrow and a pressed state, because three testers had separately told me they could not tell what was clickable. Every item got a reply in the console, including the two where the thing a tester wanted already existed and my reply told them where to find it. Version 9 followed on 9 September, the ninth release on the testing track.

I rewrote the application to one rule: cite only what Google can verify on its own side. Dated releases, release notes, feedback items and my replies to them are all events inside Google's system, while numbers from my own database, however careful, verify nothing for a reviewer. Each written answer had to fit in 300 characters, which is a discipline of its own.

I applied on 10 September, and the approval arrived on 12 September. On 14 September I sent the first production release for review, and it was live about two and a half hours later, in 177 countries. The French and Turkish listings followed the next day.

What changed, and what I can't claim

The strangest part of the whole story sits in Play's own chart. The round Google approved had fewer people opening the app than the round it rejected. From 5 to 21 August, Play counted between 4 and 10 daily users, most days between 5 and 9. From 22 August until the day I applied, it counted between 1 and 5, partly because I had told my testers they no longer needed to open it every day.

So more engagement is not what changed. What changed is where the evidence lived: feedback inside Google's own channel with a reply under every item, releases that answered specific feedback within two days, and an application that cited nothing Google could not check for itself. I believe that is what made the difference. I cannot prove it, and Google has not said.

If you are stuck behind the same requirement, this is what I would take from my five months:

1. The reviewer judges what Play Console shows. Effort that leaves no trace there does not count.

2. Play cannot see inside a Trusted Web Activity. It counts opens.

3. Describing feedback in the application is not the same as Google seeing it. Ask testers for private feedback on Play with a specific question, and reply to every item.

4. A web deploy is invisible to Play. A new bundle on the testing track is the update it sees.

5. Paying for testers bought me one tester who ever moved money, and a block of installs that sat unopened in my statistics for weeks.

As for how the launch is going, it is too early to say anything honest. Akluma has been on Google Play for a week, the public listing shows 10+ downloads, and Play's own statistics run days behind. What I can say is that it is there, in English, French and Turkish, and that it is the same app the web has been running for over a year.

Akluma is on Google Play and at akluma.com if you would like to set a goal and see whether the plan holds.

This is the first post in a short series on getting Akluma onto Google Play, and the next four each take one finding from this story further. For the background, I did keyword research for my savings app. Then I actually looked at the search results. explains why a web app needed a store listing at all, and The Savings Plan That Rewrites Itself covers what the app does.

Turning down the sensible option

The sensible option was Meilisearch. It was on the table, seriously — an open-source faceted search engine that hands you multilingual stemming and typo tolerance for free. It lost the decision for a reason I stand behind: the parts of search I most wanted to build — the ranking, the scoring, the ability to explain a result's position — are exactly the parts Meilisearch would have absorbed into a black box, and it would have added a service to a stack that otherwise fits in a single SQLite file. Also considered and rejected: FTS5 alone (throws away the structured filters the relational schema gives me) and building an inverted index from scratch in PHP (two to three weeks of reinventing what FTS5 does well, with no extra learning in the parts that matter). What shipped is a hybrid: SQL owns the filters, FTS5 owns the free text, and a scorer in PHP owns the ranking.

Two layers, one query

Every search resolves through two layers that compose. Layer 1 is plain SQL over indexed columns — country, language, years of experience, role taxonomy, tech stack, canonical-only, not-polluted — and it always runs, cutting the 13,000-listing corpus down to a slice before any text matching happens. Layer 2 is SQLite's FTS5 virtual table, which indexes exactly three columns:

CREATE VIRTUAL TABLE listings_fts USING fts5(
    title,             -- short, high-signal
    description_text,  -- the full HTML-stripped body
    indexed_org_name,  -- the employer
    tokenize = 'trigram'
);

A worked example from the calibration notes: a candidate filtered to Germany, English-speaking, at most five years of experience, software roles, types postgres backend. Layer 1 narrows 13,158 records to roughly 600. Layer 2 runs MATCH 'postgres backend' over those 600 and returns about 30 with BM25 scores. The scorer ranks the 30. No cluster, no sync job, no second store — the search index lives in the same database file as the rows it indexes.

Betting on trigrams

The tokenizer choice is where the corpus pushed back on the defaults. About half the listings are in German, and German welds its words together — Postbote is Post + Bote, and a candidate who types bote is asking a question that word-boundary tokenizers cannot answer. In a head-to-head spike on 2,000 records, FTS5's default unicode61 tokenizer found 0 matches for bote and 0 for kube; the trigram tokenizer — which indexes overlapping three-character windows and does not care where words begin — found 477 and 29. That substring property covers compound nouns and partial tech terms in one move, and it is script-agnostic, so Greek and Czech and Polish queries work with zero per-language code.

The bet has a bill attached. The trigram index is 2.35× the size of the default. There is no typo tolerance — developr returns nothing, a limitation documented and accepted rather than papered over. And a three-letter query like "SAP" happily matches inside "ASAP", "WhatsApp", and "Hansaplast", because trigrams have no concept of a word boundary. That last one grew teeth after launch: word-bounded matching eventually came back as a second FTS index (unicode61 alongside trigram), blended in one query, with a rank bonus for word-bounded hits — because a user searching "Git" deserves better than every listing containing "digital".

A title is worth twenty descriptions

Searching descriptions is what makes Postbote findable, and it is also where the noise lives: for a common word like "Manager", description-only matches run about 80% irrelevant. The answer is not to stop searching descriptions — it is to make the ranking opinionated about where a match landed. FTS5's bm25() accepts per-column weights:

ORDER BY bm25(listings_fts, 20.0, 1.0, 5.0)
-- title ×20, description ×1, org name ×5

A title match outweighs a description match twenty to one. Jobs actually titled "React Developer" rank far above jobs that mention React once in a nice-to-have list, while the description-only Postbote matches still surface below anything title-relevant. The held-out term analysis that justified this found that tech terms almost never appear in titles at all — a title-only search would lose 77–98% of real matches per term — so the weighting keeps both worlds: recall from descriptions, precision from titles.

Eight signals, and a scorer that shows its work

BM25 is one voice in the final ranking, not the verdict. The composite scorer in PHP computes eight normalized components per result: the free-text score, recency (an exponential decay anchored to the corpus snapshot's newest record, not to "now" — the product is honest about being a snapshot), description quality, a canonical-cluster bonus, role match graded by classifier confidence, language match, tech-stack match, and a visa-sponsorship signal where unknown gets a neutral 0.5 rather than a punishment. The weights live in one config file, and PHP is the deliberate seam: scoring pushed into SQL would lock it to the substrate, pushed into FTS5 would lock it to BM25.

Alongside every score, the scorer emits a per-component breakdown — weighted contribution, raw value, and a human-readable note like "posted 12 days ago" or "en+de required, you speak en" — designed to power an explainer panel where any result can answer "why this rank?". Writing this post forced an honest audit: the breakdown is computed on every request, unit-tested, and consumed by a benchmarking command — and no UI component renders it yet. The panel is now a filed issue rather than a quiet assumption, which is its own small lesson about the distance between an architecture document and a screen.

What launch taught

The biggest post-launch fix was in how typed queries become FTS5 expressions. The first version joined words with an implicit AND, which turned multi-word queries into word-bag noise — every listing containing both words anywhere. The replacement is two modes selected by syntax: a plain multi-word query becomes a phrase-or-proximity search, "data engineer" OR NEAR("data" "engineer", 100), and a comma becomes an explicit opt-in to keyword mode where each comma-separated chunk must match as its own phrase. Depending on the query, that dropped 22–98% of the noise.

The companion decision matters as much: when a query finds nothing, the engine does not silently degrade to sloppier matching to avoid an empty screen. It shows the empty state — and makes it smart. laravel developer can match zero listings as text while the corpus is full of Laravel jobs, because the tech extractor understands concepts that exact tokens miss; so the empty state offers clickable recovery tips — "try the Tech: Laravel filter (n jobs)", "try comma-separated keywords" — instead of quietly showing worse results. Honest zero over engagement-driven noise.

Would embeddings make this better? Probably — "postgres" never matching "PostgreSQL DBA" is a real gap, and semantic search sits in the V2 notes next to German morphology and learned per-user weights. But the version that exists ranks thirteen thousand listings in two dozen languages on a database engine most people call a toy, and every interesting decision in it — the tokenizer, the column weights, the eight signals, the refusal to fake results — is mine to explain rather than a vendor's to hide.

Earlier posts from the same project: The Job Finder That Reads Every Listing covers the pipeline that turns raw postings into the structured corpus this engine searches, and Fuzzy Search Without Elasticsearch is this post's sibling: the same do-it-with-the-database philosophy applied to a restaurant's client book.

This is a write-up of how that search box works. The short version is that there is no Elasticsearch in it, no Meilisearch, no Algolia, nothing that would show up on an architecture diagram as a box labelled "search". There is PostgreSQL, which the application already had, and there is a few hundred lines of TypeScript, and between them they handle typos, missing diacritics, phone fragments, email addresses, and reservation notes, and they rank what they find honestly enough to show the score to the person searching.

One box, five kinds of query

The first design decision was that staff get a single search box, not a form with fields. Mid-service nobody wants to decide whether they are searching by name or by phone; they want to type the thing they have and get the person they mean. So the very first thing the backend does with a query is guess what kind of thing it is looking at.

"ayse yilmaz"   → multi_word   (contains a space)
"ayse"          → single_word
"a.k@gmail.com" → email        (contains @)
"0533 694 12"   → phone        (digits, spaces, dashes, parens)
"1284"          → id           (digits only)

Each kind gets its own path through the system. An ID is an exact lookup and short-circuits with full confidence. An email goes through PostgreSQL's similarity() function, because emails are typed from memory and memory is fuzzy. A phone number is stripped down to bare digits — and relieved of the +90 country code, since half the staff type it and half do not — and then matched by containment in both directions, so the last five digits of a number are as good a key as the whole thing. Names, the hard case, get the full treatment.

Casting a wide net in Postgres

The name path has two layers, and the split between them is the actual architecture of the feature: the database is responsible for recall — finding every row that might plausibly be the one — and the application is responsible for precision, deciding which candidate deserves to be first. PostgreSQL's pg_trgm extension does the recall. It breaks strings into overlapping three-letter fragments and calls two strings similar when their fragments overlap enough, which is exactly the kind of matching that shrugs off a missing letter or a swapped pair.

But trigrams alone do not solve Turkish. ş and s produce different trigrams, so ayse still would not reliably reach Ayşe. The fix is to fold the alphabet down before the comparison ever happens, inside the query itself:

WHERE TRANSLATE(LOWER(name || ' ' || surname), 'şğıöüç', 'sgiouç')
    % TRANSLATE(:query, 'şğıöüç', 'sgiouç')

TRANSLATE maps the Turkish letters onto their plain ASCII neighbours — ş to s, ğ to g, the dotless ı to i — on both sides of the % operator, so the trigram comparison happens in a flattened alphabet where ayse and Ayşe are finally the same word. The same folding exists as a small character map on the TypeScript side, because the application layer is about to re-score everything the database returns and both layers need to agree on what "the same letter" means.

The database query is deliberately generous. It matches against first name, surname, both concatenations in both orders, and the free-text note on the client record, and it returns up to a hundred candidates. Generosity is cheap at this scale — a restaurant's client book is thousands of rows, not millions — and every false positive it lets through is somebody the next layer gets to quietly rank into oblivion.

Ranking in TypeScript

The hundred candidates come back into Node, and a small scorer takes over. It computes four signals for each candidate and blends them with fixed weights:

similarity =
    wordMatchScore        * 0.3 +   // query words found in name words
    wordSimilarity        * 0.2 +   // character-level agreement, word by word
    consecutiveMatchScore * 0.3 +   // longest run of matching characters
    noteSimilarity        * 0.2;    // Levenshtein distance against the note

Each signal earns its place with a different failure mode in mind. The word-match score forgives word order, so yilmaz ayse still finds Ayşe Yılmaz. The consecutive-match score rewards long unbroken runs of agreement, which is what separates a genuine near-miss from a coincidental scatter of shared letters. And the note similarity — a full Levenshtein distance computed against the free-text note on the record — exists because staff remember people by their stories: the note might say window table, birthday in May, and "birthday" is what the waiter remembers at the till. The reservation search runs the same shape of pipeline with its own weights — names carry 60 or 70 percent of the score there, notes twenty, phone ten — joins through to the clients on each booking, and refuses to return anything scoring under ten percent.

The scored list gets sorted, the top ten survive, and that is the entire ranking engine. There is no index to keep warm, no cluster to monitor, and no second data store that can drift out of sync with the truth — the search runs against the same PostgreSQL rows the booking form writes.

Showing the score to the user

One decision in the UI still pleases me: the search results show their similarity score. A result card for a client says, in small text under the phone number, similarity: 87%, and cards above seventy percent get visually marked as strong matches. Search results usually present themselves with unearned confidence — the first result simply is the answer, however weak the match was. Telling a waiter "this is 87% likely who you mean" respects both the waiter and the limits of the algorithm: a strong match gets tapped without a second look, and a page of fifty-percenters tells them to ask the caller to spell the name. The search input is debounced, results arrive grouped into a clients section and a reservations section with counts on each, and the whole exchange usually fits inside the pause where the caller says "Yılmaz. Y-ı-l…"

The extension that had to move

The production wrinkle, because there is always one: pg_trgm is an extension, and on a managed PostgreSQL instance you do not get to assume where it lands. The migration history tells the story in two files. V72_install_pg_trgm.sql is one hopeful line, CREATE EXTENSION IF NOT EXISTS pg_trgm. V73_prod_db_enable_pg_trgm_becaschema.sql exists because that was not enough on the hosted database — it drops the extension and reinstalls it with SCHEMA becaschema, pinning it into the application's own schema so the % operator and similarity() are actually reachable from the queries that need them. On a laptop, where everything lives on the default search path, the problem is invisible; it only introduces itself in production.

-- V73: the do-over
DROP EXTENSION IF EXISTS pg_trgm;
CREATE EXTENSION pg_trgm SCHEMA becaschema;

Would this design survive a million clients? Not as written — scoring a hundred candidates in JavaScript per keystroke is a luxury of restaurant-sized data, and at some point the ranking would need to move into SQL with a proper GIN trigram index behind it. But that is precisely the point. The problem was restaurant-sized, so the solution is too: one extension the database already knew how to install, one scorer small enough to read in a sitting, and a search box that finds Ayşe no matter which alphabet the keyboard thinks it is speaking.

This is the second write-up from the same project: The Restaurant Floor That Rearranged Itself covers the live floor plan that regrouped and recoloured its own tables mid-service.

A related write-up from a different project: Ranked Search on SQLite applies the same do-it-with-the-database instinct at a larger scale, ranking 13,000 multilingual job listings with SQLite FTS5 and a scorer in PHP rather than a search cluster.

docker exec <container> php artisan migrate
# SQLSTATE[HY000] [1045] Access denied for user 'app'@'%' (using password: NO)

The password is not wrong. The application beside you is using it perfectly well. It is simply invisible to you, and the reason it is invisible says something quietly important about how containers actually work.

A container does not have one environment

The setup that leads here is a good one, and an increasingly common one. Rather than leave secrets sitting in a file on disk, you have the container fetch them when it boots. An entrypoint script runs first, pulls the database password and everything else from a secrets manager, exports them into its own environment, and then hands control to the application. From that point on the application carries the password in memory and never needs a file for it. It is a clean pattern, and it is exactly what sets the trap.

The instinct most of us carry is that a container has an environment, a single bag of variables that everything inside it shares. That is not how it works. On Linux, every process carries its own copy of the environment, handed down from its parent at the moment it is created. When the entrypoint exported the password, it placed it into one process, the one that went on to become the application. Those variables live in that process and in no other.

Now consider what docker exec actually does. It does not open a window into the application's process. It starts a fresh process inside the container, and that new process is assembled from the container's original configuration, not from whatever the entrypoint exported at runtime. Your shell is a sibling of the application rather than a view into it. It inherited none of the secrets the entrypoint added after boot, so when your migration reaches for the database password, it finds nothing there. The application can see the password. You, standing right next to it, cannot.

You do not have to take my word for it, because you can see the hole directly. The application is running with that password in hand, yet ask your own shell inside the very same container to print it, and you get nothing back.

docker exec <container> printenv DB_PASSWORD
# (prints nothing)

That empty line is the whole problem, and it is true no matter what the application does with the password afterwards. The value is exactly where it should be, in the process that fetched it. It simply was never in yours.

Reaching into /proc for what you cannot see

Once you know where the secret actually lives, the way to reach it becomes obvious. Linux publishes the environment of every running process through the /proc filesystem, and the file /proc/1/environ holds the environment of process one, your application, as a list of variables separated by null bytes. If you can read that file, you can load the application's own environment into your shell and run your command with the exact values the application is using.

The single detail to get right is that the entries are separated by null characters rather than by newlines, so you read them with that in mind.

docker exec <container> bash -c '
  while IFS= read -r -d "" var; do export "$var"; done < /proc/1/environ
  php artisan migrate
'

The loop walks the null-separated list, exports each variable into the shell, and then runs the migration with the environment the application already has. Nothing is typed, nothing is written to a file, and nothing leaves the machine. The secret is copied from one process straight into another, which is the entire point. You never had to know the password in order to use it.

Do not paper over it with -e

There is a tempting shortcut at this point, and it is worth warning against, because it fails in the worst possible way. Since docker exec accepts an -e flag for setting a variable, you might reach for it and simply hand the password across yourself.

docker exec -e DB_PASSWORD='the-password' <container> php artisan migrate

This has two problems. The smaller one is that it undoes the very thing you set out to do. You went to real trouble to keep the secret off disk and out of sight, and here you are typing it onto a command line, into your shell history, in plain view of anyone glancing at your screen.

The larger problem is that your shell reads the value before docker exec ever receives it. If the password contains a character the shell treats as special, the value that finally arrives inside the container is not the value you typed. An exclamation mark can trigger history expansion. An asterisk can expand into a list of file names. The command usually does not error when this happens. It runs with a quietly wrong value, and that is the real danger. Suppose the command had not been a migration but the one that seeds your administrator accounts. Each account would be created with a password that is subtly not the password anyone was handed. Nothing would complain. The next morning the login page would turn everyone away, and there would be no error anywhere to explain why.

The lesson is not to quote more carefully. It is to keep the secret off the command line entirely. Take it from the process that already holds it, the way the earlier snippet does, and the shell never gets the chance to touch it.

None of this is exotic once you have seen it, which is exactly why it is worth writing down. The whole surprise rests on one assumption that most of us carry without ever examining it, that a container has a single shared environment. It does not. It has one environment for every process, and a secret injected at startup lives only in the process that received it. So when you need that secret from the outside, the move is not to fetch the value and type it back in. It is to reach into /proc and borrow it from the process that already holds it.

More field notes from production: The 504 That Was Actually Working, a gateway timeout on deploy day where nothing was actually broken, and The Endpoint That Moved While I Wasn't Looking, an API that had quietly relocated.

The interesting part is what happens when the plan changes. A savings tracker, drawn on a whiteboard, looks trivial. Add money, watch the number rise. But a savings plan is a promise to your future self, and promises drift. You miss a week. Your situation improves and you want to save faster. The date you were aiming for slips. The moment any of that happens, the tidy schedule you started with is wrong — and the real work of the app begins, quietly, behind a single button, where the user never sees it.

Akluma is also the project that has taught me the most, precisely because I have kept it running for real users over a long time. The lessons in this post are the kind you only learn by maintaining something, not by building it once and walking away.

Five numbers that look alike and aren't

To see how the plan rewrites itself, you first have to meet five numbers Akluma keeps on every goal, all of them columns on a single MySQL table. Three of them never change. Two of them do. I confused them often enough, early on, that I eventually wrote myself a note so I would stop.

frozen at creation:
  Price          what the thing you want costs
  Planned total  everything you plan to deposit
  Original Goal  the end total if you follow the plan
alive afterwards:
  Updated Goal   empty until the goal actually moves
  Still to save  recomputed after every change

Three numbers are frozen the moment you create the goal: the price of the thing you are saving for, the planned total of all your deposits, and your Original Goal — the end total you will reach if you follow the plan to the last payment. Those never move again. They are the promise as you first made it, kept on the record exactly as it was.

The other two are alive. One is simply how much you still have to save, recomputed after every deposit and every change. The subtle one is the Updated Goal. It stays empty — literally null in the database — and only takes a value when a recalculation genuinely moves the goalposts. If you merely re-pace the same goal, paying it off faster without changing the total, the Updated Goal stays empty and the screen keeps showing your original one. That sounds pedantic. It is actually the difference between an interface that tells the truth and one that quietly lies to you about what you set out to do.

When the goal moves

So what happens when you change the plan? Say you decide to raise your weekly deposit. You press recalculate, and a Laravel service runs the entire change inside a single MySQL transaction.

First it checks that the change makes sense. The goal is still active. The new amount is positive. There are payments left to reschedule. The new amount is smaller than what you have left to save. Then comes the decision I am most pleased with. The old pending payments are not deleted. They are archived — flagged, set aside, kept on file.

you raise your weekly deposit:
  old pending payments   →  archived (v1), not deleted
  new schedule           →  written fresh (v2)
  payments already made  →  left untouched
  ( one transaction: all of it commits, or none does )

I chose a plain archived flag over Laravel's soft deletes for a reason I wrote down at the time: I wanted to preserve the old schedule as an audit trail. When a user asks "why did my plan change?", the answer should still exist. With the old payments set aside, the service writes a fresh schedule and stamps every new payment with an incremented version number. Version one was your original plan. Version two is this one. The versions let me trace a goal's whole history, generation by generation, which has rescued me more than once when something looked wrong in production.

Only then does the app decide whether your goal truly moved, fill in the Updated Goal if it did, and recompute what you have left. If anything fails along the way, the transaction rolls back and your old plan is still standing, untouched. The schedule rewrites itself, but there is never a half-rewritten moment in between.

The bug the plan didn't predict

Here is the kind of thing you only find by living with your own code. When I planned this feature, I wrote down that the new payments should keep numbering from the last payment the user had actually completed. It seemed obvious. Continue from where they left off.

It was wrong. Because the old pending payments are archived rather than deleted, they keep their numbers, sitting quietly in the table. Numbering a new schedule from the last completed payment would have collided with those archived rows. So the shipped code does something slightly different from my plan. It takes the highest payment number across every row, archived ones included, and counts up from there. A one-line change, with a comment to my future self explaining why. The plan did not predict it. The running system did.

A smaller discipline runs underneath all of this too. Money is never a float in Akluma. Every amount passes through Brick\Money, a dedicated PHP money library that keeps currency and precision honest. That matters more than usual when six currencies are in play and two of them — the African francs — have no decimal places at all. A rounding shortcut that looks harmless in dollars quietly breaks in francs.

What the project demonstrates

I want to be honest about Akluma's scale. It is a live web application with a few dozen real users, my own savings among them. There is an Android version, but it is a thin wrapper around the website rather than a native app, and it is still in testing — not yet on the Play Store. The reminders that nudge you to save are real, and they go out by email at nine in the morning in each user's own timezone; the push-notification version is designed but not yet built. I would rather tell you exactly what is shipped than dress it up.

None of the stack is exotic, and that is the point. Akluma is a Laravel 12 and PHP application. The front end is Blade and Alpine.js on Tailwind CSS, built with Vite. The data lives in MySQL, the money runs through Brick\Money, and a Pest test suite guards the parts that matter. It deploys to Fly.io on every push through GitHub Actions, across separate staging and production environments. These are boring, proven tools — chosen so I can keep trusting them for years.

What the project demonstrates is not scale. It is care. The engineering worth showing lives in the boring, invisible places. A plan that can change without losing its history. A set of financial numbers that know which of them are allowed to move. A transaction that refuses to leave a half-finished mess. A small bug fixed because I was still around to notice it. Those are the things that decide whether software can be trusted with something as personal as a person's money.

The hard part of a savings app is not adding up money. It is letting a plan change while keeping faith with the plan it used to be — and being honest, on the screen, about which promise you are still keeping.

Akluma is live at akluma.com if you would like to set a goal and watch it hold.

Related: building the plan was only half the work. The other half was getting the right people to find it. I wrote that story separately: I did keyword research for my savings app. Then I actually looked at the search results.

Part of a short series of write-ups on the things I've built. See also The Restaurant Floor That Rearranged Itself, a reservation system that moved its own tables, and The Job Finder That Reads Every Listing, a job search that reads every listing with local language models.

The part that isn't the booking form

The restaurant ran the way most do, in sittings: an early evening seating and a later one, a Sunday breakfast, an open buffet, each with its own window of hours and its own rhythm. A host taking bookings over the phone holds the whole room in their head — which tables can be pushed together for the big group at eight, which corner empties out in time for the couple at half past nine, which section is closed tonight — and the quiet ambition of the software was to hold that same picture without ever asking a human to redraw it by hand. The booking form, the customer records, the statistics the owner liked to look at, all of that was real work but it was ordinary work; the thing that turned the project from a calendar into a system was the floor plan, and the floor plan only earned its keep once it could change on its own.

The floor as a living data model

The application was a full-stack Nuxt 3 build, a Vue front end talking to a Nitro server in Node, with everything persisted through Prisma into PostgreSQL, and the first real design decision was how to make a physical room addressable. Every table carried two positions rather than one: a home position, where it sat when the floor was at rest, and a live position, where it actually stood at this moment in the middle of service — and that small duplication, a default_location it could always be returned to and a current_location that drifted as the night wore on, turned out to be the hinge the whole feature swung on.

one table on the floor:
  table_no          12
  shape             "round"
  default_location  (540, 300)   ← home
  current_location  (580, 285)   ← now, mid-service
  colour            the seated reservation’s status

The tables belonged to named sections — a garden bar, a main salon, a living room with a fireplace — and the front end drew them onto a floor plan as circles and rectangles sized to their real dimensions, each one filled with a colour it did not choose for itself. The colour came instead from whatever reservation currently owned the table, read off a status whose label could be something as plainly human as Gelmedi, "not arrived", so that a glance at the screen was meant to tell you the same thing a glance across the room would: this table is empty, this one is seated, this one is still waiting on its guests.

Rearranging the floor every half hour

This is the heart of it. Every half hour a timer would wake, ask the database which reservations were live at that precise moment, and for each party that had been promised more than one table it would take the lowest-numbered table as an anchor and walk the others out from it in a patient diagonal — forty pixels across and fifteen up for each table in the group — committing the whole rearrangement in a single transaction, so that the floor never flickered through a half-moved state with tables caught mid-stride between where they had been and where they now belonged.

a party of ten, three tables pushed together:
  table 1 the anchor — stays where it is
  table 2 +40 across, −15 up
  table 3 +80 across, −30 up
  ( one transaction, then pushed live to every screen )

What I found quietly satisfying about the design is that there was never a "group" in the database at all. A group was not an entity you created and destroyed; it was simply the fact of several tables sharing a single reservation, made visible by the coordinates the sweep computed for them, and so it could come into being and dissolve again without anything ever needing to be explicitly torn down. A companion sweep, running on its own half-hour offset, did the dissolving: it looked for any table standing away from its home position that no longer had a live reservation to justify the move, and it sent each of those quietly back to where it belonged. Building and testing this against real clock time was the hardest part of the entire project, because the logic only made sense in motion — a table is grouped because a party is seated now, and ungrouped because that now has passed — and getting it to behave correctly meant reasoning constantly about the exact moment the code believed it was living in.

Keeping every screen in agreement

A restaurant floor on a busy night is not run by one person but by several at once — a host at the door, waiters threading between the tables, someone watching the whole room from a screen at the back — and the one thing the system could not afford was for two of those screens to disagree, so every move, every reset, and every newly written reservation went out over a WebSocket the moment it happened, and every other screen quietly redrew itself to match, until the floor on the glass and the floor in the room were telling the same story. The server announced each change on its own named channel, a tablesMoved when the half-hour sweep regrouped a party, a tablesReset when the companion sweep sent tables home, and the front end answered each announcement the same way: it went back to the database, fetched the floor afresh, and recomputed every colour, so that no client was ever left quietly reasoning from a stale picture of the room.

What the project demonstrates

Stepping back, what the project really shows is a single physical reality — a restaurant floor that hosts already knew how to manage by hand — modelled faithfully enough in software that it could manage itself: a full-stack Nuxt and Node application over PostgreSQL, a floor plan rendered from live data, scheduled jobs that reshaped that floor on a clock, and a real-time layer that kept every screen honest about it. There is an honest coda I owe the work, too: the system was finished and deployed, but it never actually opened to a single diner, because the restaurant's appetite for adopting it ran into business questions that had nothing to do with whether the software worked, and I have made my peace with the fact that a thing can be built well and still never get its night in service.

The most interesting engineering in a reservation system is not the reservation; it is everything the restaurant already knew how to do by hand — pushing tables together, pulling them apart, reading the room by colour — quietly taught to a database and made to keep itself honest every half hour.

What I keep from it is less a feature than a habit of mind: that the surest way to make software genuinely useful to people who work with their hands is to model the world they already live in faithfully enough that the screen and the room never have to be reconciled by a person.

A second write-up from this same project: Fuzzy Search Without Elasticsearch covers the search box that finds a client called Ayşe when a hurried waiter types ayse, using PostgreSQL trigrams and a small scoring layer instead of a search cluster.

Part of a short series of write-ups on the things I've built. See also The Savings Plan That Rewrites Itself, a savings app that rewrites its own plan, and The Job Finder That Reads Every Listing, a job search that reads every listing with local language models.

The jobs you can actually apply to

Picture the person this is built for, because every decision in the project flows from them: someone looking to move to Europe for work, often from a developing country and often with limited time and energy, who opens a board of thirteen thousand listings and needs, with as little ceremony as possible, the small handful of roles they could realistically apply to tomorrow. The trouble is that the obstacles standing between that person and those roles are almost never written into the structured fields a job board lets you filter on; they are buried in the description text, in whatever language the employer happened to write it in. A posting might be written entirely in German for a candidate who does not speak it, or quietly demand five years of experience the candidate has not yet earned, or refuse visa sponsorship in a single sentence three paragraphs down, or be built around a technology stack the candidate has never once touched — four mismatches that decide everything, and not one of which is reliably available as a filter, because not one of them lives anywhere but in the prose.

Reading every listing before ranking it

So the heart of the project is a Python pipeline whose entire job is to read each posting and convert that prose into structured signals a search engine can actually use, and the design principle underneath it is one I kept returning to: lean on cheap, deterministic heuristics for the cases that are clear, and reserve a language model for the genuinely ambiguous minority that heuristics quietly get wrong. The clear cases really are clear — a title that says "Backend Engineer", a requirements list that names "Python, PostgreSQL, AWS" under its own header, a German posting that insists on verhandlungssicheres Deutsch (business-fluent German) — and for those a well-built regex pass is faster, more reproducible, and frankly more honest than any model would be. It is the remaining slice that needs judgment, the posting where English is wünschenswert (desirable) rather than required, or where a residency requirement hides halfway down a paragraph about benefits, and that slice is precisely where the models earn their keep.

And the models that handle those hard cases all run locally, on my own machine: every one of them is an open-weights model — Llama 3.1 and Gemma 4 — served through Ollama rather than called through a paid API, which means the pipeline can read the entire corpus as many times over as a calibration pass demands without a budget meter ticking in the background, and it means the same code runs identically whether or not there is a network in front of it. What comes out the other end, for a single posting, is a compact structured row that existed nowhere in the source:

one German-language posting  →
{
  languages_required:  ["de", "en"],
  experience_years:    3,
  role:                "Backend Engineering",
  tech_stack:          ["Java", "Spring Boot", "PostgreSQL"],
  sponsorship:         "none_stated",
  relocation:          "provided"
}

What makes a row like that worth the effort is that nearly every field had to be inferred across a language barrier rather than copied from a field that already existed, and the pipeline does exactly this across the roughly two dozen languages the corpus actually contains — German, French, Spanish, Italian, Dutch, Polish, Czech, Hungarian, and a long multilingual tail behind them — applying the same heuristics-first, model-for-the-hard-cases discipline to language detection, years-of-experience extraction, role classification, tech-stack tagging, and the visa-and-relocation signals that matter most to the very person the whole thing is being built for.

Calibrated against hand-labelled truth, not vibes

The temptation with this kind of extraction is to glance at a few outputs, decide they look about right, and ship, and the discipline I held myself to instead was to measure every extractor against a set of postings I had labelled by hand, drawn deliberately from outside the set I used to tune it — because grading a thing on the same examples you tuned it on is the oldest way there is of lying to yourself about how well it works. The numbers that came out of that exercise are the ones I am proudest of, precisely because they are real rather than flattering: language detection lands at a hundred percent precision and ninety-seven percent recall against its held-out labels; the years-of-experience extractor, after a cleanup pass, reaches a hundred percent precision and ninety-three percent recall; the visa, sponsorship, and relocation signals catch every genuinely positive case in the held-out set, a clean hundred percent recall on the one axis where a miss would actually cost the candidate a job they could have had; and the tech-stack tagger sits at roughly three-quarters precision while getting its required-versus-nice-to-have distinction right about eighty-two percent of the time.

The role taxonomy was the most interesting piece of machine learning in the project and also the one I am most careful never to oversell. Rather than impose a set of categories from the top down, I embedded the job titles with a multilingual embedding model called bge-m3 and clustered them, letting the natural job families fall out of the data into a taxonomy of twenty-two top-level functions that I then went through and validated by hand. Classifying a brand-new title into that taxonomy is genuinely hard, and so I treat the model's guess for exactly what it is — a soft signal that helps rank and group the results rather than a hard filter that hides anything — because a classifier that is right about a role two times out of three has no business silently deciding what a job-seeker is never allowed to see.

The app, the search engine, and one read-only file

All of that signal would sit inert without something to deliver it, and the front half of the project is a Laravel 13 application with a Vue and Inertia frontend, built on top of a search engine I had to write rather than install. The free-text search runs on SQLite's full-text engine, FTS5, but settling on a single tokenizer turned out to be a trap: the substring-friendly trigram tokenizer is wonderful for German compound words, surfacing Postbote (mail carrier) when you type "bote" and Kubernetes when you type "kube", and it is quietly catastrophic for short names, returning a page full of Digital Marketing roles the moment a candidate searches for "Git" because the letters g-i-t happen to live inside the word "digital". The fix was to run a second, word-level index alongside the first and blend their rankings, so that whole-word matches rise cleanly to the top while the substring bridges remain available underneath them rather than drowning everything above them:

q = "Git"
before:  0 of the top 5 results were real  (Digital Marketing, Chief Digital Officer …)
after:   5 of 5 were real Git / version-control roles

Sitting on top of retrieval is a composite relevance scorer that weighs the free-text match against recency, description quality, role and language and tech-stack fit, and a small bonus for the canonical entry of a duplicate cluster — duplicates being a very real problem on a board where one logistics template can recur, byte for byte, across hundreds of cities, which is why a deduplication pass collapses roughly forty percent of the corpus before the candidate ever lays eyes on it. Calibrating that scorer against a hand-labelled set of queries lifted its ranking quality, measured as NDCG at ten, from 0.69 to 0.83 on a held-out set I had sealed away during the tuning. The whole thing then ships to Fly.io as a single read-only SQLite file, a little over a gigabyte, baked directly into the deploy image, which holds the running cost to roughly six dollars a month and makes a data refresh nothing more exotic than another deploy.

What the project demonstrates

Stepping back from the individual parts, what this project is meant to show is a single arc owned end to end: a multilingual data-and-machine-learning pipeline in Python that orchestrates local open-source language models to turn unstructured prose into calibrated, measurable signal, and a full-stack Laravel and Vue application with a custom hybrid search engine that turns that signal into something a real person can actually use, deployed and running in production for the price of a sandwich. The same rule runs through every layer of the project: I never trust a model just because it sounds confident.

Applied-AI engineering, done honestly, is far less about clever prompting than it is about knowing exactly how often you are wrong — and being willing to hand the hard cases to a model only once you have proven, against labels you wrote by hand, that it has earned the trust.

The job finder is live at jobs.ubeyd.dev if you would like to apply your own filters and watch a noisy board collapse down to the roles you could genuinely apply to.

Related: The 504 That Was Actually Working — a deeper dive on one challenge from this project, the deploy-day gateway timeout where nothing was actually broken and the fix was a cache that can never be wrong.

Also from this project: Ranked Search on SQLite picks up where this post stops, covering the search engine itself: FTS5 with a trigram index, title-boosted BM25, and an eight-signal scorer in PHP.

Part of a short series of write-ups on the things I've built. See also The Restaurant Floor That Rearranged Itself, a reservation system that moved its own tables, and The Savings Plan That Rewrites Itself, a savings app that rewrites its own plan.

What resolved was a 504 Gateway Time-out, in the flat grey of an error page that has stopped trying.

A timeout with a healthy machine

My first move was denial, which is everyone's first move: I reloaded, and got the same 504 exactly sixty seconds later, to the second. I opened the Fly.io dashboard and the machine was sitting right there — awake, healthy, no crash loop, no out-of-memory kill, nothing in the logs with the shape of a stack trace. Every signal the platform offered me said the application was fine. It simply would not answer the door.

HTTP/2 504
server: nginx

So I SSH'd onto the running machine to interrogate it directly, and the picture got stranger rather than clearer. The database file was exactly where it should be, all 1.1 gigabytes present and accounted for. The health endpoint — the little /up route Laravel ships for precisely this kind of moment — came back instantly. The page I actually wanted did not.

GET /up   → 200   in 0.29s
GET /     → 504   at 60.00s

Those two lines are the whole mystery. The same machine could answer one route in under a third of a second and could not answer the other before nginx gave up at sixty. The lobby light was plainly on while the front door timed out. Whatever was wrong was not the machine; it was something specific to the one page I cared about.

The database had an alibi

My first real suspect was the database, and I want to be honest that it was a good suspect. SQLite's query planner leans heavily on a set of statistics that a command called ANALYZE produces, and when those statistics are missing or stale the planner will cheerfully choose the wrong index and turn a sub-hundred-millisecond query into one that takes five, ten, thirty seconds. This very project had been bitten by exactly that before. A cold machine, freshly deployed, serving a query that hangs for half a minute is precisely the silhouette that ANALYZE trouble casts — so reaching for the database was not a wild stab, it was the single most plausible explanation on the board.

I checked it the only way worth checking, by looking at whether the statistics were actually there:

sqlite> SELECT count(*) FROM sqlite_stat1;
36

Thirty-six rows of statistics, which meant ANALYZE had run during the import and the planner had everything it needed. The database had an alibi. While I was in the mood, I ruled out the other familiar ghost — server-side rendering, which runs in its own little Node process and can hang in genuinely interesting ways — by poking its port and watching it refuse the connection instantly, which is the difference between "not running" and "hanging." Two of my three best theories were dead inside ten minutes, the page was still timing out, and I was no closer to why.

183 seconds

The obvious thing I had somehow not yet done was time the page — not through nginx, which could only ever tell me "longer than sixty seconds," but underneath it, asking the application itself how long it took to assemble the response. So I dropped into Laravel's tinker shell and rendered the page by hand, twice.

1st render:  183.2s   → 200 OK
2nd render:   21.1s   → 200 OK

There it was, and it rearranged the entire problem. The page was not broken. It was returning a flawless 200 — it was just taking a hundred and eighty-three seconds to produce it the first time, and twenty-one on every visit after. Nothing had crashed. The page was thinking.

What it was thinking about was relevance. Every search runs its matching listings through a composite scorer that pulls each row into PHP and computes a score for it, one at a time, before ranking the set — and on the broadest possible query, the one a visitor sees before they have narrowed anything, that is the entire corpus, something like thirteen thousand listings, scored in a PHP loop. Twenty-one seconds of honest arithmetic, warm. The first render is so much worse because everything is cold at once: the PHP opcode cache is empty, the app's own caches are empty, and that 1.1-gigabyte SQLite file has not yet been pulled into the operating system's page cache, so even the disk reads are paying a one-time warmup tax. SQLite was in the room — just as a contributor to the cold tax, never as the bill itself.

And then the decision that turned a slow page into a broken one. To keep a hobby project cheap, I had told the Fly.io machine to scale to zero whenever it went idle. It is a lovely idea right up until you notice what it means: every visit after a quiet spell now wakes a cold machine and pays the full hundred-and-eighty-three-second tax, and nginx, not unreasonably, stops waiting at sixty. The 504 was not an error report at all. It was a stopwatch.

The cache that can't be wrong

The fix came in three parts, and only one of them is interesting. The two dull ones first: I stopped the Fly.io machine from scaling to zero, so a single instance stays warm with its opcode cache and its page cache intact, and I raised the timeouts — nginx's and PHP's both — to five minutes, so that on a genuinely cold render the page is allowed to finish rather than being cut off at sixty seconds. Together those turn the worst case from "broken" into "slow exactly once."

The interesting part is the cache. I wrapped the whole scored page in Laravel's Cache::remember, keyed on the exact filter state, so the expensive thirteen-thousand-row loop runs once per distinct view and every visit afterward is served from cache in milliseconds:

$props = app()->isProduction()
    ? Cache::remember(\(key, now()->addDay(), fn () => \)this->buildProps(...))
    : $this->buildProps(...);

And here the deployment model I picked at the very beginning quietly pays for itself. Caching is usually the easy half of a hard problem; the hard half is invalidation — knowing the precise moment a cached answer has gone stale and tearing it down. I wrote none of that, and I never will, because the data this cache is built from cannot change underneath it. The corpus is a read-only SQLite file baked into the image, and the only way the data ever changes is a new deploy — which ships a new image, a fresh machine, a fresh and empty cache. The file and the cache are replaced together, atomically, by the single act of deploying.

A cached page here isn't "probably still fresh." It is structurally incapable of being stale — because the only thing that could spoil it also throws it away.

Had a managed Postgres sat behind this app, with the application writing to it, that paragraph would be a lie, and I would owe the codebase a careful, bug-prone invalidation layer for the rest of its life. The read-only-baked-SQLite model is exactly what lets me delete that entire category of work. The database that looked so guilty in the second act turns out to be the reason the fix is clean.

What the error was telling me

I think this one cost me an extra beat because 504 reads as a crash. It lives in the same mental drawer as 500 and segfaults and the stack traces you go hunting through — the drawer labelled something is broken. But nothing here was broken. The machine was healthy, the file was intact, the query was correct, and the page returned a perfect 200 the moment you gave it enough room. The error was never a failure; it was a mismatch — between how long the page took to think and how long the proxy in front of it had agreed to wait. There was no bug in the ordinary sense to find, because the bug was a decision: I had parked a cold-expensive page behind a scale-to-zero switch, and the timeout was simply the seam where those two choices met.

The lesson I am keeping is a small and slightly embarrassing one about reading errors literally. A 504 does not mean broken. It means this took too long — and "too long" is a question with two halves, how long the work takes and how long you have agreed to wait for it, and the fix can live in either half. I spent my first hour certain the work was wrong. The work was fine. I had simply never told anyone how long it was allowed to take.

Related: The Job Finder That Reads Every Listing — the project this gateway timeout came from, an end-to-end tour of the AI pipeline that reads every listing and the app it ships in. And The Endpoint That Moved While I Wasn't Looking — another deploy-day investigation where the obvious, loudly-announced explanation turned out to be the wrong one. And The Secret Your Container Has but Your Shell Can't See, on a database password that quietly vanished inside docker exec.

Then I started a new side project, wanted to be able to blog from it too, and went to wire the same two commands into the new repository. The wiring itself was the boring kind of work: a gitignored file to hold the credentials, a directory for the drafts to live in, a couple of lines so the publish command knew where to read its API key from. None of it was interesting, and all of it worked. /draft-post wrote files. A read query against my own blog came back instantly with the right data. And then I tried to actually publish something, and the whole thing walked straight into a wall I would spend the rest of the afternoon trying to climb.

A 301 with no body

The publish step does one thing: it POSTs a GraphQL mutation to Hashnode's API endpoint. Instead of the JSON I expected, I got this:

HTTP/2 301
location: https://hashnode.com/announcements/graphql-api
server: cloudflare

The body was the small, anonymous Cloudflare stub that says 301 Moved Permanently and nothing else. It is worth sitting with how strange that response is, because the strangeness was a clue I did not read carefully enough at the time. A working API does not redirect you. It returns data, or it returns a 401, or it hands you a structured error object explaining what you did wrong. A 301 to a marketing announcement is the API declining to behave like an API at all — it is the front door of the building telling you the building has been replaced by a billboard. And the billboard had an address: /announcements/graphql-api. The redirect was, in effect, pointing at the reason it existed. I just read the wrong reason off it.

I paid for the wrong fix

The announcement, when I followed it, explained that Hashnode had retired free access to its GraphQL API a few weeks earlier. Every request — reads and writes alike — now required a Pro plan on the publication. Reads, the page noted almost in passing, used to be free; they were not anymore.

This fit the symptom so cleanly that it did not feel like a theory at all, it felt like a diagnosis. My request was being turned away, and here was an official page, which my request had been redirected to, explaining that requests now got turned away unless you paid. The redirect was practically holding up a sign with the answer on it. So I upgraded. I paid, I waited a few minutes for whatever needed to propagate to propagate, and I re-ran the exact same command — and the redirect came back identical to the byte.

That is the most dangerous kind of red herring there is: a symptom that lines up perfectly with a recent, loudly-announced change. It is plausible enough to spend money on. I want to be honest about the money here, because the honest version is more interesting than the self-deprecating one. The fifty dollars was not wasted — the API genuinely does require Pro now, so I would have needed it regardless of anything else. But it was also not the fix. I had been missing two different things at the same time, and buying Pro had quietly removed exactly one of them, which is the worst possible outcome, because it leaves the symptom completely unchanged and you with no way to tell that you made progress. My next move was to generate a brand-new access token from the now-paid account, on the reasonable theory that my old token predated the whole change. Same redirect. Not one byte different.

Three AIs and one blind spot

When I am properly stuck, I do a particular thing: I write down everything I have already ruled out, in as much detail as I can stand, and I hand that brief to a couple of other models. So I took the evidence to ChatGPT and to Gemini, alongside the Claude session I was already working in, and I laid it all out. The token did not matter — old and new both failed identically. The headers did not matter — I had tried it with and without an explicit Accept, with a real browser User-Agent glued on, no difference. The path did not matter — a trailing slash still redirected, a /graphql suffix timed out at the edge, a hostname I guessed at returned a flat 404. It was not my machine and not my country, because I had reproduced the same redirect from my laptop and from a second machine on another continent, landing on two different Cloudflare data centers, every single time.

The two models agreed with each other, and with me, on the shape of the thing. The endpoint was the documented one. Pro was supposed to switch the API on automatically. It was not a propagation delay. The redirect was happening at the edge, before authentication ever entered the picture. Where they split was on the cause, and the split is the part I keep thinking about. Gemini was confident and admirably specific: the breaking change, it told me, was the format of the Authorization header — Hashnode now expected a Bearer prefix, and the edge was bouncing anything that did not match the pattern. It cited integration write-ups to back the claim. ChatGPT was more cautious, noted that Hashnode's own documentation contradicted itself on whether the token wanted a Bearer prefix at all, and then made the observation that should have stopped me sooner than it did: a redirect that fires before authentication cannot possibly care what your token looks like.

I tested Gemini's fix anyway, because it was the only concrete lever anyone had handed me. Raw token, Bearer token, lowercase bearer token — three requests, three identical redirects. It was a beautifully constructed theory, with sources, and it was wrong the instant it met a curl.

The dashboard had it the whole time

I had a support email half-drafted and I was making my peace with publishing the next few posts by hand when I opened the Hashnode dashboard to copy the support address out of it. And there, on the first screen, in a panel I had apparently never once read, was a heading that said Your GraphQL API endpoint, and underneath it:

https://gql-beta.hashnode.com

Not gql.hashnode.com. gql-beta. The endpoint had moved. Every request I had made all afternoon — every request that the Claude session and ChatGPT and Gemini had so carefully reasoned about — had been aimed at a hostname Hashnode had quietly retired, and that retired hostname now does precisely one thing: it 301s everyone, paid or free, valid token or not, sensible headers or not, to the announcement explaining that the API had changed. The redirect had never been a paywall and never been an auth problem. It was a forwarding address on an old door.

One request to the new host:

curl -s -X POST https://gql-beta.hashnode.com \
  -H "Content-Type: application/json" \
  -d '{"query":"{ publication(host: \"ubeyd.dev\") { title } }"}'

{"data":{"publication":{"title":"ubeyd.dev"}}}

A flat 200, and the data sitting right there. A minute later a real publish mutation put a throwaway test post on the live site, and I opened it in the browser just to watch it exist. The thing that had eaten my afternoon, survived a paid subscription, and defeated three language models reasoning in concert was a four-character difference in a URL, displayed in a panel on the first screen of a dashboard I open every single time I publish.

What I'd been assuming

The uncomfortable part is not that I missed it. The uncomfortable part is that all four of us missed it in exactly the same way. The Claude session, ChatGPT, Gemini, and me — we reasoned carefully, and within the frame we had silently accepted we reasoned correctly. We ruled out tokens and headers and networks with real, methodical rigor. What not one of us did was interrogate the single thing the entire investigation was standing on: that the URL was still the right URL. The bug was not hiding in any of the variables we kept testing. It was sitting in the one constant we never thought to make a variable.

There is a sharper edge here for anyone who, like me, has started reaching for a second and third model when the first one gets stuck. Adding ChatGPT and Gemini did not help, and — this is the part worth internalizing — it could not have helped, because all three models shared the identical blind spot. They could multiply my reasoning, but they could not multiply my access. The single surface that held the answer was the product's own dashboard, behind my own login, and that was mine to go and look at and nobody else's. I had been treating "I am stuck" as a reasoning problem, to be solved by throwing more reasoning at it, when it was an access problem the whole time.

Three models can multiply your reasoning. They cannot multiply your access. The thing none of them can see is still yours to go and look at.

It rhymes, a little uncomfortably, with something I wrote here not long ago about an AI quietly building seed data that disagreed with production in ways no test could catch — that an AI collaborator can only ever be as correct as the ground truth you show it. This was that same lesson wearing a different hat. The ground truth was not in the documentation, which was stale. It was not in the changelog, which described the wrong half of the problem with great confidence. It was not in the agreement of three capable models, who were unanimous and unanimously working the wrong frame. It was on a panel I had never read, on a dashboard I look at every time I ship a post. I just had to be the one to actually look.

Related: Six Slash Commands I Built on Top of Claude Code — the two commands in this story, /draft-post and /publish-post, are two of the eight I didn't cover there. And The 504 That Was Actually Working — another deploy-day investigation where the obvious, loudly-announced explanation turned out to be wrong. And The Secret Your Container Has but Your Shell Can't See, on a database password that quietly vanished inside docker exec.

$ ls -1 ~/.claude/commands/
close-gl-issue.md
draft-post.md
gh-issue.md
gl-issue.md
handoff.md
improve-issue.md
prod-db-akluma.md
publish-post.md
ship-akluma.md
ship-aclient.md
staging-db-akluma.md
staging-db.md
update-docs.md
worktree.md

Each file is a Claude Code slash command — a markdown document whose contents are prepended to the agent's prompt when you type /<filename> into a session. They are, quite literally, just text. They do not feel like they should matter very much. They do, though, and they are the difference, for me, between a raw AI assistant that is vaguely useful on a good day and a set of sharp, purpose-built tools that absorb specific pieces of toil from a Laravel legacy codebase I did not originally write. The CLI is not really my tool. These files are.

This post is a tour of six of them, ordered loosely by the chronology of a working day. Some of the fourteen are variants for a different project, and a few are database tunnels that are useful without being interesting; I have left those out. The six I have picked are the ones whose job description surprised me the most — either when I wrote them, or later, when I caught myself realising how much of my day they had quietly eaten.

/improve-issue <url>

The first command in the catalog is also the most embarrassing. I create GitLab issues during meetings, and those meetings move fast enough that my issue descriptions are often one-third of a sentence and a typo. Sometimes the title is literally the last thing someone said out loud. I know that when I get back to my desk, I will not remember what it meant, and more importantly, the Claude Code session I eventually point at the issue will not either. So I built a command that rewrites my own bad issues into real ones before I start work on them.

It takes the full URL of an issue, sniffs the platform, fetches the issue and every comment through the appropriate CLI, and then picks between three triage paths depending on how clear the thing is. Before any of that, though, if the issue smells UI-shaped — a mention of a page, a form, a button, a layout — the command is hard-coded to stop and demand a screenshot before going any further. No exceptions. I have wasted too much time guessing which page was broken from a title like "fix asset register" to ever want the command to be polite about this again.

The output is not a comment on the issue. It is a full replacement of the title and description, posted via the API:

glab api "projects/ateam%2Faclient_docker_src/issues/<number>" \
  --method PUT -f title="\(TITLE" -f description="\)BODY"

Comments annotate; a rewritten description replaces. I want the original carelessness gone, not appended to.

/gl-issue <number>

This is the load-bearing command of the whole directory. It is the one I run most days, and the one that most obviously rewards the effort of having written the command file in the first place. The friction it absorbs is a subtle one: starting work on an issue on a legacy codebase used to mean the agent would either drown in the project's several hundred pages of documentation, reading everything and running out of context before it even touched the code, or skip the documentation entirely and implement the issue blind. Neither was a good outcome. What I wanted was selective reading — load the specific documentation categories this particular issue actually needs, and nothing else.

The way the command does this is, I think, the single nerdiest thing in the directory. When the command fires, it spawns a separate Claude subagent running on Sonnet — cheaper and faster than the main Opus session — whose only job is to read docs/doc-router.yaml, read the issue, and return a JSON array of category IDs for the main agent to load. The instruction inside the subagent's prompt is explicit about the format:

Return ONLY a JSON array of category IDs, e.g.: ["auth", "database", "tenant-isolation"]

The main agent then reads every file under each matched category plus an "always" set that loads on every issue, and only then starts investigating the codebase. The triage adds about thirty seconds to the start of a session and saves easily twenty minutes of the main agent chasing the wrong files. The command also has a hard-coded screenshot gate that fires before any code is read — if the issue involves anything visual, the agent is required to stop and ask for a screenshot before going any further — and a TDD Red-Green-Pause loop inside the implementation step, because I like my agents to write the failing test before they touch the production code. The doc triage is the part that surprised me; the rest is just the command enforcing habits I already had and kept forgetting to apply.

/worktree <number>

Parallel worktrees have their own two-post story, which I will link at the end of this one, so I will skip over the infrastructure and talk only about the command I eventually wrapped around it. Before this command existed, spinning up a new parallel session looked like this: fetch the issue, read the title, derive a short kebab-case slug by hand, pick a prefix (feature/ or bugfix/ based on the labels), assemble the branch name, run ./scripts/parallel-session.sh create feature/export-report-pdf-345 345, wait for the script to print the worktree path, and then run code ../aclient-wt-export-report-pdf/ to open the new window. Six steps, most of them mechanical and therefore forgettable.

Now it is:

/worktree 345

One integer. The command fetches the issue with glab, slugifies the title, picks the prefix from the labels, runs the script, parses the worktree path from the script's output, and opens a new VS Code window on the worktree directory. The output it hands back looks like this:

Session ready for issue #345!

  Directory:  ../aclient-wt-export-report-pdf
  App:        http://localhost:902

A new VS Code window is open. Start Claude there and run:
  /gl-issue 345

The last line is a deliberate pass-off to the next command in the pipeline. The destroy mode is the symmetrical opposite — /worktree destroy 345 finds the session by issue number, tears down the Docker stack, syncs develop, and deletes the branch — which means the whole lifecycle of a parallel session is now bookended by two commands that each take a single integer as their argument.

/handoff

This is the weird one. It is also the only command in this post that you do not use by typing a number into a fresh session — you use it by pausing the one you are already in. The friction it absorbs is a specific one: sometimes, mid-investigation, a small side-task turns up that does not really warrant a branch of its own. A one-line config change, a dependency bump, a typo I noticed in a file I was reading for a completely unrelated reason. Creating a new feature branch for a ten-line change feels heavy, and so does ignoring it and hoping I remember later. What I wanted was a way to hand the task to another Claude session — one that already has a live feature branch — and have that session fold the work into its own next commit.

The command produces a single fenced code block, nothing else, which I copy and paste into the other session's chat window. The block contains a self-contained implementation brief: investigation summary, file paths with line numbers, the exact edits, the verification steps. But the part that makes the whole thing work is the preamble of git rules that the brief opens with:

Git rules — READ CAREFULLY:
- Do NOT create a branch, switch branches, or run any git commands
- Do NOT commit this work separately — fold it into your next commit alongside your own work
- Do NOT stash, discard, or touch your existing uncommitted changes
- Do NOT question why this task is on your branch — it's intentional

That last rule is the one I had to add after the first time I tried this, because the receiving session, very reasonably, looked at the task it had just been handed, noticed that it had nothing to do with the branch it was on, and started asking me whether I was sure about this. I was. I am still. Every branch has a minimum cognitive cost — a new MR to review, a new CI run to wait for, a new context to hold open in my head — and some ten-line tasks are not worth paying that cost for. The command encodes that judgment so I do not have to re-argue it every time.

/ship-aclient push

This is the command that absorbs the git shipping workflow: commit, push, create the merge request, wait for the pipeline, merge, sync, clean up. It has four modes — push, develop, staging, and quick — and the most interesting of the four is quick, because of the guardrail built into it.

The problem the guardrail solves is that I trust myself to run quick for small, low-risk changes, and I do not trust myself to know the difference. A typo fix is obviously quick. A routing refactor is obviously not. Somewhere between the two there is a long tail of changes where my judgment is wrong often enough to matter. So the command does not trust me either. Before it proceeds with a quick flow, it checks the diff against two criteria. The first is a hard-coded list of high-risk file patterns:

Dockerfile, docker-compose.yml, docker/**, deploy/**
.gitlab-ci.yml, composer.lock, package-lock.json
routes/web.php, config/**, app/Http/Middleware/**
app/Providers/**, app/Services/**, database/migrations/**

If any file in the diff matches any of those patterns, the command refuses to proceed quietly and asks me to confirm that I really do want to take the quick route. The second criterion is a numeric threshold — four files and a hundred changed lines, calculated from the project's eightieth-percentile commit size — with an exclusion list of always-low-risk paths like resources/views, tests, and docs that do not count toward the total. Between the two criteria, the command catches essentially every change that my past self would have regretted shipping on the quick route.

/update-docs

If /gl-issue is the load-bearing command of the catalog, /update-docs is the most quietly valuable one. I run it after every issue I close, and roughly half the time it does something I did not expect. The obvious half of its job is the expected one: read the documentation index, compare it against the diff, and update any docs that have drifted out of sync with the code. That is the UPDATE action in the command's plan output. The less obvious half is an action called CAPTURE, which is governed by a step inside the command file called Knowledge Audit.

The step asks a single question: if a fresh LLM session had to work on the same area of code tomorrow, what would it waste significant time re-discovering? Then it applies a filter — the knowledge has to have taken more than a few minutes to figure out, it has to not be obvious from reading the code casually, and it has to not already exist in any documentation the command has just read. Anything that survives the filter is written out as a short markdown note and saved into a codebase-notes/ subfolder, named after the area of code rather than after the issue — data-inventory-normalization.md, not issue-309-notes.md. The template for these notes is deliberately sparse:

# <Area of Code>

## What This Covers
<one-sentence summary>

## Key Findings
### <finding title>
<what it is, why it matters, where it lives>

## Files Involved
- `path/to/file.php` — role in this flow

Those notes are, in effect, a knowledge base I am building for my own future agents. Every time a fresh session starts work on an area I have been in before, it can read the note that the previous session left behind and skip the part where it re-discovers, for the third or fourth time, that the double-query pattern in FormsController is there because of a legacy French translation fallback. The command does not replace my human memory of the project — the project is too large for that to be a realistic goal — but it gives my agents a working memory that survives between sessions, which is a thing the raw tool does not offer.

Six commands, one pattern

None of these commands is individually clever. Each one started as a single frustration I had hit in a single session, and each one got written the way a sysadmin writes an alias — to stop having to type the same thing twice. What they have in common is not the specific instructions inside each file, which are mostly unremarkable, but the shape of the thing they are doing underneath. Each command absorbs one specific piece of cognitive toil that I used to carry in my head and am quietly relieved not to carry anymore. The CLI is not my tool. These files are. There are eight more in the directory I did not write about, and I would miss every one of them if they vanished tomorrow.

Related: How I Run Parallel AI Coding Sessions on the Same Laravel Project and The half I left out of my parallel worktrees post — the two earlier posts in this loose series on building tooling around Claude Code for a Laravel legacy project.

The empty worktree

The quieter problem was that each freshly spawned worktree was a ghost town. Running migrate:fresh --seed would populate the database with countries, question templates, rating scales, and a handful of starter users, and that was all. Structurally, it was a complete application. Functionally, every page beyond the admin panel was an empty table. There were no filled-out forms, no assets, no incidents, no remediation plans, and no way to tell whether a visual change to the audit register was actually working without first clicking through ten minutes of form-filling to generate something for the register to display. That rather defeats the point of being able to spin up three sessions in ninety seconds.

What I really needed was a seeder that produced a populated application. Not just the reference data that every install needs — the lookup tables and the question templates — but the kind of transactional data that a real tenant would accumulate after a week of actual use. Finished assessments. Audits with ratings and the remediation plans those ratings imply. A handful of incidents. Assets with real map coordinates so the location map would actually render. The obvious move was to build Laravel model factories for every domain model I cared about and then compose them inside a single DemoDataSeeder that stitched them together into a believable scenario.

Factories and a seeder

That is what I set out to do, and it went more or less the way you would expect. Working with Claude, I built the factories in dependency order — starting with the simple lookup tables, then users and organizations, then the structure of the forms, then the form instances, then the assets, then the responses, and finally the remediation plans and incidents that hang off everything else. Each factory got a small smoke test that asserted it could create a single instance and persist it without anything blowing up. The repository's test count went from eighty-nine to one hundred and nine, and every page I cared about in the application started showing data again. The dashboards had numbers. The audit register had two completed audits. The asset register had twelve assets, four of them with map pins. The application looked like a functioning product for the first time in my worktrees.

The shape of the result I was aiming for was a single method chain that would do most of the setup work for me — something like this:

User::factory()->organization()->create();

One line, and behind the scenes it would create a new organization account, attach the fifteen rows of reference data that every real organization in this application needs in order to function, and leave me with an object I could then hang assets, audits, and responses off of inside the demo seeder. Getting a single line to do that much useful work is really what all twenty-or-so factories were for.

One small design decision is worth mentioning, because it turned out to make a disproportionate difference to how pleasant the environment was to use. My original plan had been to create separate demo organizations — Acme Corp, Beta Industries — each with their own fictitious admin users, so that you could log in as demo-orgadmin@example.com and see their data. I prototyped this, and then I threw it away. Remembering a new set of credentials every time you spin up a worktree is friction, and friction compounds when you are spinning up worktrees all day long. The better approach was to hang the demo data off the Test Organization that already existed in the bootstrap seeders, so that the familiar orgadmin@example.com account — the one I use everywhere else — would quietly find itself in front of a fully populated application with no new accounts, no new passwords, and no mental overhead at all.

The final piece was the environment guard. I wanted the seeder to run on every environment that wasn't production — local development for me, but also our staging server, where the team needed a populated application for the same reasons I did. One line inside DatabaseSeeder was enough:

if (! app()->environment('production')) {
    $this->call(DemoDataSeeder::class);
}

A normal deployment runs php artisan migrate, which doesn't touch seeders at all, so shipping this code to staging was entirely safe. The demo data only appears when someone explicitly runs migrate:fresh --seed, which is exactly when — and only when — you want it to.

One more check, before I merged

At this point I thought I was done. The factories existed, the seeder ran, the application was populated, and the test suite was green. I was ready to open the merge request and move on. Before I did, though, I wanted to do one more check — the kind of check that doesn't feel necessary until you have been bitten by skipping it. I wanted to compare what my seeder was actually putting into the database against what a real user had been writing into our staging environment over the previous few weeks. Not the shape of the data, not the row counts, but the exact column values, row by row and field by field.

I am very glad I did, because my seeder was lying to me, and the lies were subtle enough that no automated test would ever have caught them.

A column called user_email

The first mismatch I found is still, several days later, my favourite. My seeder was populating a column called user_email on internal audit responses with, reasonably enough, the email address of the user who had submitted the response. The real application, it turned out, writes the literal string '0' into that column for internal users, and nothing in the schema, nothing in the column name, and nothing in any reasonable person's mental model would ever have hinted at this.

What my seeder wrote:    user_email = 'orgadmin@example.com'
What production writes:  user_email = '0'

It was one of those small pieces of behaviour that only exists because a controller, at some point years ago, needed a sentinel value and nobody ever revisited the decision. My seeder was not wrong in a way that would break anything — the application was perfectly happy with 'orgadmin@example.com' sitting in that column. It was wrong in a way that made the seeded data look slightly nicer than the real data, and that, I am now convinced, is the worst kind of wrong, because it makes local development quietly drift away from production reality without anybody noticing.

There were ten of these mismatches in total, and every single one of them traced back to the same underlying pattern. The controllers in this application write their data using raw SQL INSERT statements that only touch a specific subset of columns — nine, in some cases, out of a table with thirty — and every other column in the row is left sitting at whatever default the migration originally set, which was almost always NULL. A client ID my seeder was cheerfully populating. A progress percentage my seeder was calculating. A type field my seeder was deriving from the form. None of these fields were actually written by the application in production. All of them had, quietly and without any particular announcement, become columns that the schema promised but the code simply never touched.

Correcting the seeder, once I understood the pattern, was the easy part. I went back through the DemoDataSeeder and stopped setting any column that the application itself did not set, and the staging diff came back clean. The harder question — and I think the reason this whole detour is worth writing about at all — is how I would ever have caught any of this without the staging comparison. I do not think I would have. Every automated test I could have written would have been reading the same schema I was writing to, which means the tests would have agreed with the seeder that everything was in order. The only authoritative source of truth was the production code path itself — the actual controller writing the actual INSERT statement, and the actual staging database that had been touched by that code path for weeks on end. Anything less than that would have missed it.

What I actually learned

This, I think, is the lesson that matters most, and it is specifically a lesson about working with an AI collaborator on a legacy codebase. Claude was a very capable partner throughout this work. It built the factories in the right order. It wrote smoke tests without being asked. It raised sensible questions about ownership columns and prerequisite chains and the difference between assessment responses and audit responses, and it caught several mistakes I would otherwise have made on my own. What it could not do — and I do not think any AI collaborator could reasonably have done — was know that user_email = '0' was the right value. That knowledge does not live in the schema, it does not live in the column names, it does not live in the tests, and it does not live in any piece of documentation I had put into the conversation. It lives only in the line of controller code that writes the INSERT, and in the staging database that reflects the consequences of that line running against real users over time. If I had not gone looking for it, neither of us would have found it.

So the moral of this follow-up, for me, is a slightly adjusted version of the one I ended the first post with. That first post was about teaching the AI about the environment — making sure it understood that a worktree was not an ordinary branch, and making sure it would not run cleanup commands that would kill the session. This one is about something adjacent, and a little more uncomfortable. It is about accepting that on a legacy codebase, an AI collaborator can only ever be as correct as the ground truth you show it.

The schema is not ground truth. The code that actually runs in production is.

If you are going to let the AI write your seed data, you have to be willing to compare what it wrote against a database that was shaped by that production code — and you have to be willing to do the comparison yourself, because nothing else will do it for you.

Where things stand now

The parallel worktrees, for what it is worth, are now as genuinely useful as I had hoped they would be when I wrote the first post. I can spin up three sessions at once, and every single one of them boots into an application that looks and behaves like a real product. The infrastructure, it turns out, was the easy half of the problem. The data — and the small, uncomfortable process of making sure the data wasn't quietly lying to me — was the other half, the half I didn't write about the first time round. This is me writing about it now.

Related: How I Run Parallel AI Coding Sessions on the Same Laravel Project — the first post in this pair, where the whole parallel-worktree setup got built in the first place.

I wanted to launch a second session on a different issue. And a third. Each one visible in the browser. Each one isolated. Each one disposable.

The problem? Docker Compose is bound to one directory. One project. One set of ports. You can't just docker-compose up twice and hope for the best.

Here's how I solved it.

My setup: macOS, VS Code with the Claude Code extension, Docker Desktop, and a Laravel 12 app running in Docker Compose. Everything in this post assumes a similar stack, but the core ideas — parameterized ports, isolated databases, worktree-aware tooling — apply to any Dockerized web application.

The problem in one sentence

Docker Compose ties your localhost:8000 (or whatever your app port is) to a single project directory. If you want to work on two branches simultaneously and see both in the browser, you need two separate Docker stacks on different ports — and they can't collide.

Why not just use git stash?

Because stashing is sequential. I don't want to:

1. Stash my work
2. Switch branches
3. Spin up the other feature
4. Verify it
5. Switch back
6. Pop the stash
7. Remember where I was

I want to open two VS Code windows, each on a different branch, each with its own localhost URL. Work on both. Alt-tab between them. No context switching tax.

The architecture: worktrees + parameterized Docker

The solution has two parts:

1. Git worktrees give you multiple checked-out copies of the same repo. They share the git object store (so they're lightweight — not full clones), but each has its own branch, working directory, and files.

~/dev/
├── my-project/                    # Main repo (localhost:8000)
├── my-project-wt-export-button/   # Worktree 1 (localhost:8001)
└── my-project-wt-fix-sidebar/     # Worktree 2 (localhost:8002)

2. Parameterized Docker Compose ports let each directory run its own stack without port collisions.

Step 1: Make your ports configurable

Replace hardcoded ports in docker-compose.yml with environment variables:

services:
  app:
    ports:
      - "${APP_PORT:-8000}:80"
  mysql:
    ports:
      - "${DB_PORT:-3306}:3306"
  mailpit:
    ports:
      - "${MAIL_HTTP_PORT:-8025}:8025"
      - "${MAIL_SMTP_PORT:-1025}:1025"

The :-8000 syntax means "use 8000 if the variable isn't set." Your main repo works exactly as before — zero behavior change. But now each worktree can set its own ports in a local .env.

Step 2: Isolate session cookies

This one bit me during testing. I logged into localhost:8000, then localhost:8001, then went back to :8000 — and I was logged out.

Why? Browsers share cookies across ports on the same domain. Both apps set a cookie called laravel_session for localhost. Logging into one overwrites the other.

The fix: each worktree gets a unique session cookie name.

# In the worktree's .env:
SESSION_COOKIE=my_app_session_8001

Laravel reads SESSION_COOKIE from the environment. One line, problem solved.

Step 3: Automate everything with a script

I wrote a ~200 line bash script that handles the entire lifecycle:

# Create a parallel session
./scripts/parallel-session.sh create feature/export-button
# Output:
#   Session ready!
#   App: http://localhost:8001
#   MySQL: localhost:3307
#   Mailpit: http://localhost:8026

# List active sessions
./scripts/parallel-session.sh list
# Output:
#   my-project-wt-export-button  http://localhost:8001  [feature/export-button]  (running)

# Destroy when done
./scripts/parallel-session.sh destroy feature/export-button

The create command:

1. Runs git worktree add to create the worktree (from the develop branch)
2. Copies .env from the main repo
3. Overwrites port variables with auto-calculated unique values
4. Sets a unique SESSION_COOKIE
5. Runs docker-compose up -d

Port assignment is simple: session N gets base port + N. If you destroy session 1 and create a new one, it reuses the gap.

| Session     | App    | MySQL  | Mailpit |
|-------------|--------|--------|---------|
| Main repo   | :8000  | :3306  | :8025   |
| Worktree 1  | :8001  | :3307  | :8026   |
| Worktree 2  | :8002  | :3308  | :8027   |

Step 4: Seed independently

Each worktree gets its own MySQL volume. The database starts empty. After creating a session:

cd ~/dev/my-project-wt-export-button
docker-compose exec app php artisan migrate:fresh --seed

Timing matters here. If your Dockerfile uses a named volume for vendor/, it starts empty on first boot. Your entrypoint likely runs composer install to populate it. You need to wait for that to finish before running artisan commands — otherwise you'll get "class not found" errors because the packages aren't installed yet. My script waits for Apache to respond (which means the entrypoint completed) before attempting to seed.

This is actually a feature, not a bug. Each session has a clean database, seeded from scratch. No stale data from another branch's migrations. No "works on my machine because I manually ran that one migration."

The daily workflow

Starting a parallel session:

1. Run the create script (30 seconds — Docker images are cached)
2. Run migrate:fresh --seed (10 seconds)
3. Open the worktree directory in VS Code
4. Start working

While working:

• Each VS Code window is a completely independent project
• Each has its own localhost URL for browser testing
• Git works normally — commit, push, create PRs/MRs
• Tests target the correct containers automatically (Docker Compose resolves from the current directory)

Cleanup after merging:

Once you've merged the MR in GitLab, the cleanup flow is straightforward:

1. Close the worktree's VS Code window — there's nothing left to do there.
2. Switch to the main repo's VS Code window and run the destroy command:

./scripts/parallel-session.sh destroy feature/export-button

This stops the Docker containers, removes the database volume, deletes the worktree directory, and leaves you back on develop. From there you can sync your local branch with a quick git pull origin develop and delete the feature branch with git branch -d feature/export-button. The entire teardown takes about five seconds.

"But what about database conflicts?"

This was my biggest concern too. If session 1 adds column pdf_path and session 2 adds column export_format to the same table, what happens?

Nothing dramatic. Each session creates a migration file with a unique timestamp. When both branches merge:

• Both migration files coexist in database/migrations/
• php artisan migrate runs them in timestamp order
• Both columns get added

The worktree databases are disposable scratch pads. You never merge database data — you merge migration files. The reconciliation happens in git, the same way it always does.

The only real conflict scenario: two branches modify the exact same column. That shows up as a git merge conflict in the PR, same as if you'd built the features sequentially.

Resource cost

Each parallel session runs its own MySQL, Redis, and app container. On my M-series Mac:

• ~250MB RAM per session
• ~500MB disk per MySQL volume
• Docker image layers are shared (cached), so no extra disk for the app image

I comfortably run 2-3 sessions alongside my main repo. Beyond that, my laptop would probably start complaining.

What I'd do differently

Cookie isolation should be the default. I wasted 20 minutes debugging mysterious logouts before realizing cookies were the issue. If you're building this, add SESSION_COOKIE to your .env.example from day one.

Named volumes are your friend. Docker Compose auto-prefixes volume names with the project name (derived from the directory name). Each worktree directory has a unique name, so volumes are automatically isolated. I didn't have to configure anything — it just worked.

Don't share the database. I briefly considered having all worktrees connect to the same MySQL instance (different databases). Bad idea. Different branches have different migrations. Isolation is the whole point.

Don't race your own entrypoint. My first attempt added a composer install step to the setup script — but the container's entrypoint was already running composer install to populate the empty vendor volume. Two concurrent composer installs on the same directory corrupted zip downloads and crashed the container. The fix was to let the entrypoint do its job and have the script wait for Apache to respond before proceeding.

Check your API key restrictions. If you use third-party APIs with domain or URL restrictions (Google Maps, Stripe, OAuth callbacks), they're probably locked to localhost:8000. Your worktree on localhost:8001 will get silent failures — a blank map, a rejected OAuth redirect, a 403 from a payment form. I got a console error on Google Maps because my API key's "Website restrictions" only listed :8000. The fix was trivial (add :8001/* and :8002/* to the allowed list), but the debugging wasn't obvious. Add your worktree ports to every restricted API key upfront.

Audit every git add -A in your tooling. If you symlink shared directories into worktrees (documentation, assets, anything gitignored), a blanket git add -A can pick them up in unexpected ways. I now require explicit file-by-file staging in worktree contexts. It's a minor inconvenience that eliminates a whole category of "why did that get committed?" moments.

Teach your AI agent about worktrees

This is the part nobody warns you about. Your Docker setup can be flawless, your ports perfectly isolated, your databases independently seeded — and your AI coding assistant will still trip over itself if it doesn't know it's running inside a worktree.

The first time I shipped code from a worktree session, the agent finished by cheerfully suggesting: "When you've merged, run /ship-aclient develop to sync up." That command switches to the develop branch — which is already checked out in the main repo. Git refuses to check out the same branch in two worktrees. The command fails immediately.

The fix was to teach the agent to detect its environment. If the working directory name contains aclient-wt-, the agent knows it's in a worktree session and adjusts its behavior accordingly. Instead of suggesting branch switches and local cleanup, it tells you to close the VS Code window and run the destroy command from the main repo. Same outcome, correct path to get there.

The second issue was subtler. I symlink the project's docs/ directory into each worktree so the agent can reference documentation during implementation. The directory is gitignored, so it should be invisible to version control. But git add -A — the lazy, catch-all staging command that every AI agent defaults to — doesn't always play nicely with symlinks in worktree contexts. The solution was to forbid git add -A entirely when working in a worktree and require the agent to stage files by name. It's a small constraint that prevents a class of confusing errors.

If you're building a similar setup with an AI coding tool, the lesson is this: the infrastructure is the easy part. The hard part is making sure your agent's playbook — its prompt templates, command files, or system instructions — includes conditional logic for the worktree context. Anywhere your prompts say "switch to develop," "sync the branch," or "stage all changes," you need a worktree-aware variant. Otherwise your carefully isolated sessions will work perfectly right up until the moment the AI tries to clean up after itself.

The result

I went from working on one issue at a time to running 2-3 parallel AI sessions. While Claude is writing tests for feature A, I'm reviewing the implementation plan for feature B. While waiting for feature B's CI pipeline, I'm verifying feature C in the browser.

My throughput roughly doubled. Not because the AI got faster, but because I stopped being the bottleneck.

The entire setup is:

• One docker-compose.yml change (4 hardcoded ports to 4 env variables)
• One .env.example update (document the new variables)
• One bash script (~200 lines)
• One line in .gitignore (docker-compose.override.yml)

No external tools. No paid services. No complex orchestration. Just git, Docker, and a shell script.

Follow-up: The half I left out of my parallel worktrees post — how I populated each worktree with realistic data, and the staging comparison that caught my demo seeder quietly lying to me.

The code snippets in this post are simplified from a real production setup. Your ports, service names, and seeder commands will differ. The principles are universal for any Dockerized web app.