The Secret Your Container Has but Your Shell Can't See

Here is a small mystery that cost me an afternoon the first time I met it, and that I have watched trip up good engineers since. You have a containerised application running in production. It holds a database password, because it fetched one when it started, and it is serving requests without complaint. Then you need to run a single command against it by hand, a database migration, say. You open a shell into the running container and run the migration the ordinary way, and it fails at once with an error that makes no sense.

docker exec <container> php artisan migrate
# SQLSTATE[HY000] [1045] Access denied for user 'app'@'%' (using password: NO)

The password is not wrong. The application beside you is using it perfectly well. It is simply invisible to you, and the reason it is invisible says something quietly important about how containers actually work.

A container does not have one environment

The setup that leads here is a good one, and an increasingly common one. Rather than leave secrets sitting in a file on disk, you have the container fetch them when it boots. An entrypoint script runs first, pulls the database password and everything else from a secrets manager, exports them into its own environment, and then hands control to the application. From that point on the application carries the password in memory and never needs a file for it. It is a clean pattern, and it is exactly what sets the trap.

The instinct most of us carry is that a container has an environment, a single bag of variables that everything inside it shares. That is not how it works. On Linux, every process carries its own copy of the environment, handed down from its parent at the moment it is created. When the entrypoint exported the password, it placed it into one process, the one that went on to become the application. Those variables live in that process and in no other.

Now consider what docker exec actually does. It does not open a window into the application's process. It starts a fresh process inside the container, and that new process is assembled from the container's original configuration, not from whatever the entrypoint exported at runtime. Your shell is a sibling of the application rather than a view into it. It inherited none of the secrets the entrypoint added after boot, so when your migration reaches for the database password, it finds nothing there. The application can see the password. You, standing right next to it, cannot.

You do not have to take my word for it, because you can see the hole directly. The application is running with that password in hand, yet ask your own shell inside the very same container to print it, and you get nothing back.

docker exec <container> printenv DB_PASSWORD
# (prints nothing)

That empty line is the whole problem, and it is true no matter what the application does with the password afterwards. The value is exactly where it should be, in the process that fetched it. It simply was never in yours.

Reaching into /proc for what you cannot see

Once you know where the secret actually lives, the way to reach it becomes obvious. Linux publishes the environment of every running process through the /proc filesystem, and the file /proc/1/environ holds the environment of process one, your application, as a list of variables separated by null bytes. If you can read that file, you can load the application's own environment into your shell and run your command with the exact values the application is using.

The single detail to get right is that the entries are separated by null characters rather than by newlines, so you read them with that in mind.

docker exec <container> bash -c '
  while IFS= read -r -d "" var; do export "$var"; done < /proc/1/environ
  php artisan migrate
'

The loop walks the null-separated list, exports each variable into the shell, and then runs the migration with the environment the application already has. Nothing is typed, nothing is written to a file, and nothing leaves the machine. The secret is copied from one process straight into another, which is the entire point. You never had to know the password in order to use it.

Do not paper over it with -e

There is a tempting shortcut at this point, and it is worth warning against, because it fails in the worst possible way. Since docker exec accepts an -e flag for setting a variable, you might reach for it and simply hand the password across yourself.

docker exec -e DB_PASSWORD='the-password' <container> php artisan migrate

This has two problems. The smaller one is that it undoes the very thing you set out to do. You went to real trouble to keep the secret off disk and out of sight, and here you are typing it onto a command line, into your shell history, in plain view of anyone glancing at your screen.

The larger problem is that your shell reads the value before docker exec ever receives it. If the password contains a character the shell treats as special, the value that finally arrives inside the container is not the value you typed. An exclamation mark can trigger history expansion. An asterisk can expand into a list of file names. The command usually does not error when this happens. It runs with a quietly wrong value, and that is the real danger. Suppose the command had not been a migration but the one that seeds your administrator accounts. Each account would be created with a password that is subtly not the password anyone was handed. Nothing would complain. The next morning the login page would turn everyone away, and there would be no error anywhere to explain why.

The lesson is not to quote more carefully. It is to keep the secret off the command line entirely. Take it from the process that already holds it, the way the earlier snippet does, and the shell never gets the chance to touch it.

None of this is exotic once you have seen it, which is exactly why it is worth writing down. The whole surprise rests on one assumption that most of us carry without ever examining it, that a container has a single shared environment. It does not. It has one environment for every process, and a secret injected at startup lives only in the process that received it. So when you need that secret from the outside, the move is not to fetch the value and type it back in. It is to reach into /proc and borrow it from the process that already holds it.

---

More field notes from production: The 504 That Was Actually Working, a gateway timeout on deploy day where nothing was actually broken, and The Endpoint That Moved While I Wasn't Looking, an API that had quietly relocated.