> ## Content Index
> Fetch the complete content index at: https://til.codes/llms.txt
> Use this file to discover other available public pages before exploring further.

# When the Nix Eval Cache Serves You a Ghost Derivation
- URL: https://til.codes/when-the-nix-eval-cache-serves-you-a-ghost-derivation/
- Published: 2026-05-31T22:00:57.000Z
- Updated: 2026-05-31T22:00:57.000Z
- Description: Postgres kept booting on port 15433, but my config said 15432. I grepped the entire repo for 15433 and found nothing. The number didn’t exist in any file I had edited, yet there it was, running, confident, every single time.
- Author: Manu Ajith
- Tags: nix, postgresql, devenv

The project config said `15432`. The process was on `15433`. Tests could not connect, because they were dialing the port the app expected and nothing was answering there.

I opened `devenv.nix`. Port 15432\. I opened `.env`. Port 15432\. I grepped the whole repo for 15433\. Nothing. Not a single reference. And yet `devenv up` had booted Postgres on 15433, and it had done it confidently, with no warning, every single time.

So the question was simple and infuriating. Neither `.env` nor `devenv.nix` mentions 15433 anywhere. Both say 15432\. Where is Postgres getting 15433 from?

One caveat before blaming the cache: in devenv 2.x, the Postgres port is also the base for automatic port allocation, so 15433 can be legitimate if 15432 is already occupied. That did not make the mismatch less real. It just meant I needed to find the exact generated config the running process had loaded.

## What is actually listening

First, confirm reality. The config is a claim; the running process is the fact. So I went and looked at the fact.

The Postgres that was actually serving had its data directory at `.devenv/state/postgres`, and that directory had a `postgresql.conf` in it. Line 17:

```
port = 15433

```

There it is. Not in any file I had edited. Sitting in the runtime state directory that devenv manages. So the next question writes itself. Who put 15433 in `.devenv/state/postgres/postgresql.conf`?

## Walking the process tree

In this setup, devenv did not start Postgres directly. It started a process manager wrapper, which ran a chain of generated scripts. Following the running process back up:

```
devenv-processes-postgres
  -> /nix/store/vy3ndzy...-start-postgres/bin/start-postgres
    -> /nix/store/f7zr5mg...-setup-postgres/bin/setup-postgres

```

`setup-postgres` is the interesting one. It is the script that prepares `$PGDATA` before the server boots. And the way devenv ships a `postgresql.conf` is not by templating it into the state directory at runtime. It bakes the config into the Nix store at evaluation time and then copies it into place. The relevant line inside that setup script:

```
cp /nix/store/dasdnbj41...-postgresql.conf "$PGDATA/postgresql.conf"

```

So the `postgresql.conf` in my state directory was not authored by me and was not derived from the `devenv.nix` I had open. It was `cp`\-ed out of a specific store path. Read that store path:

```
$ cat /nix/store/dasdnbj41rjsw5hnp94vw2vfdllqvz20-postgresql.conf
...
port = 15433

```

The store object had 15433 baked in. The lie was now located. It lived in an immutable Nix store path, and `setup-postgres` faithfully stamped it into my data directory every time it ran.

## The part where it gets genuinely weird

Here is the catch. My current shell's PATH did not point at that derivation.

When I checked which `start-postgres` was on my PATH, it resolved to a *different* store hash:

```
/nix/store/90bj99hi3f1d1vffwl34qbag44f15043-start-postgres/bin/start-postgres

```

But the process actually running had been launched from:

```
/nix/store/vy3ndzyhaa961vd5swbahdlx0hzafi0w-start-postgres/bin/start-postgres

```

Two different `start-postgres` derivations. The one in my shell would copy a 15432 conf. The one that the live process had actually come from was an older build that copied the 15433 conf.

Same name. Different content. My environment was internally inconsistent: the interactive shell knew about the new world, but the running daemon was still a fossil from the old one. That alone did not prove the eval cache was guilty; it proved the live process was not using the derivation I thought it was using. That is why grepping the repo found nothing. The 15433 was never in the repo. It was frozen inside a store object that the live process was still pointed at.

Wait. If `devenv.nix` already says 15432, why was the old derivation still getting revived?

## The eval cache is the part to distrust

devenv keeps an evaluation cache so it does not have to re-run Nix evaluation on every command. It lives here:

```
.devenv/nix-eval-cache.db
.devenv/nix-eval-cache.db-shm
.devenv/nix-eval-cache.db-wal

```

That cache maps evaluated attributes to the files, environment variables, and options they depended on. It is supposed to invalidate when a source file read during evaluation changes. Fast in the common case. A trap when that dependency tracking goes sideways.

The cache should have noticed my edit and rebuilt the process config. But the result I kept getting was the stale `start-postgres` derivation, and forcing the cache to refresh was what finally broke the loop. That old derivation referenced the stale `postgresql.conf` store path. `setup-postgres` copied 15433 into my data directory. Postgres booted on 15433\. Every restart reproduced the exact same wrong port, which is precisely why it felt like gaslighting. A flaky bug you can dismiss. A bug that is perfectly consistent and contradicts the file in front of you is the one that makes you doubt your own eyes.

## The fix

To get unblocked for tests immediately, the cheapest move is to stop arguing with the running process and just point the client at where Postgres actually is. Set the port in `.env` to match reality:

```
DB_PORT=15433

```

Postgres was already up on 15433, so tests connect right away. One line, reversible, buys you time.

But that is a workaround, not a fix. The actual repair is to force devenv to refresh its eval cache so it re-reads the file I edited:

```bash
# stop the devenv processes first
devenv up --refresh-eval-cache

```

If that still leaves the cache wedged, the sledgehammer version is deleting `.devenv/nix-eval-cache.db` and its SQLite sidecar files, if they exist.

With the cache refreshed, devenv re-evaluates `devenv.nix`, produces a fresh `start-postgres` derivation, that derivation references a `postgresql.conf` store path with `port = 15432`, and `setup-postgres` copies the correct conf into `$PGDATA`. Postgres comes up on 15432\. The config and the process finally agree, and I reverted the `.env` line.

There is a tempting third option I deliberately avoided: hand-editing `.devenv/state/postgres/postgresql.conf` from 15433 back to 15432\. It works until the next `devenv up`, at which point `setup-postgres` cheerfully `cp`s the stale store conf right back over your edit. Editing the copy is pointless when the source of the copy is the thing that is wrong.

The Nix lesson is the one worth carrying out of this. There are three layers and you have to keep them straight: the config you declare in `devenv.nix`, the immutable artifact that config evaluates to in the store, and the eval cache that is supposed to decide when to re-derive that artifact. When the running process diverges from the file you are editing, the file is rarely the liar. Walk down to the store path the process actually loaded, account for port allocation, and if that path is older than your last edit, suspect the process state and the cache that handed it to you.