I pushed an API key file to main, and the fix was mostly boring habits
A blanket git add committed a credentials file I had dropped next to my compose files. What happened, why my secret check missed it, and how I stage commits now.
The commit was called "ChatOps v2". Twelve files, 559 insertions, and somewhere in the middle of the stat output one line that had no business being there:
.gitignore | 5 +
UPDATING.md | 9 +-
kavita/OPDS_KEY | 2 +
ntfy/enable-chatops.sh | 33 ----
ops/README.md | 113 +++++++-----
ops/chatops.env.example | 13 +-
ops/chatops.py | 332 +++++++++++++++++++++++++++---------
ops/ntfy-private/docker-compose.yml | 35 ++++
ops/rollback.sh | 20 +++
ops/selfhosted-chatops.service | 9 +-
ops/setup-chatops.sh | 53 ++++++
ops/test_chatops.py | 109 ++++++++++++
12 files changed, 559 insertions(+), 174 deletions(-)kavita/OPDS_KEY is a two-line file I had created earlier that day. It held the API key for my Kavita book server and the OPDS address with that key embedded in it. I put it in the app folder because that was the folder I was working in, and I wanted it somewhere I could find it again. It went into the commit, the commit went to main, and main went to the remote. Thirty-six seconds later a second commit removed it. The gap between those two commits is what this post is about, along with the hour before them and the two days after, when the same kind of mistake happened again in a different shape.
Nothing bad came of it as far as I know. The repo is private, I regenerated the key in Kavita, and the old one is dead. But the file is still in the history, and I chose to leave it there. I'll get to why.
How the repo is used
Some context, because the shape of the workflow is what made this easy to do.
Everything I self-host lives in one git repo on my laptop: one folder per app, each with a docker-compose.yml, an .env.example, often a setup.sh that generates secrets, sometimes a README. The VPS has a clone of the same repo. I deploy by SSHing in, running git pull, then docker compose up -d in whatever folder changed. There is no CI for deploys, no pipeline, no secrets manager. The real .env files exist only on the server and are git-ignored.
Here is Karakeep's setup.sh, complete, with only the hostname changed:
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"
[ -e .env ] && { echo ".env already exists — nothing to do"; exit 1; } # never overwrite live secrets
rand() { head -c 512 /dev/urandom | tr -dc 'A-Za-z0-9' | head -c "$1"; }
mkdir -p data meili
umask 077 # .env ends up mode 600
cat > .env <<ENV
KARAKEEP_HOST=karakeep.example.com
KARAKEEP_VERSION=release
NEXTAUTH_SECRET=$(rand 48)
MEILI_MASTER_KEY=$(rand 48)
NEXTAUTH_URL=https://karakeep.example.com
KARAKEEP_DISABLE_SIGNUPS=false
ENV
echo "Done. docker compose up -d, register at https://karakeep.example.com, then set KARAKEEP_DISABLE_SIGNUPS=true."That pattern is the reason the working tree is full of things that must never be committed. Secrets get generated where the compose file can see them, which is right next to the code.
Most of the editing in October 2026 was done by an AI coding assistant working in that laptop repo. It writes the compose files and scripts, runs whatever checks it can, commits and pushes. I pull and deploy. That split comes with its own lessons (it can't reach Docker, for one), which I've written up separately. Here the relevant part is simpler: at some point I told it, in so many words, "commit and push directly to main bro :)". I meant it. I didn't want a PR flow for a one-person repo. That message is also how the key ended up in the remote, so the smiley has aged a bit.
The other relevant detail is that the working tree is never clean. I drop things into it. Over that week the app folders collected, at various times, an rclone config with a Google service-account JSON, a NocoBase API key in nocobase/.api-key, the ChatOps environment file with a TOTP secret in it, and the Kavita key file. Some of those were already covered by .gitignore. Some were not.
The first blanket add worked, which was the problem
The very first commit of this repo, back on the 7th, was made with git init && git add -A && git commit. It was fine. The .gitignore had been written a minute earlier, it excluded .env, */data/ and a folder I didn't want published, and the commit contained exactly what it should. The assistant even checked afterwards that nothing from the excluded folder had been committed.
So git add -A went into the "works" bucket in my head. A blanket add is only as safe as the ignore file at that moment, and the ignore file only knows the names someone thought of in advance. Which is fine until someone invents a new name.
On the morning of the 14th, a big batch went in: ntfy, Uptime Kuma, Authelia, Dozzle, Karakeep, FreshRSS and Kavita, all in one commit. Before committing, the assistant ran git status --short, looked at a list that was mostly ?? lines for whole new app folders, and checked the staged changes for secrets. It reported back that it had looked for .env, rclone.conf, users_database.yml and API keys, and found none. That check was real and it was useful. It was also a list of names. Anything not on the list wasn't a secret, as far as the check was concerned.
There's a subtler problem with that morning's listing too. git status --short collapses an untracked directory into a single line: ?? kavita/ tells you nothing about which files are inside it. The first commit of a new app folder is exactly when a stray file is most likely to be in there, and exactly when the short status hides it. git status --short --untracked-files=all lists every file, and that's the flag I use now.
The hour before the commit
The part I find most instructive is that the key file was spotted before it leaked.
Around 21:34 my time I pasted a long message into the session: some questions about Kavita, a docker-compose.yml from the server, the output of a status script, and a mention of the key file I'd created. The assistant's very next line was that it would first check whether that key file could end up in git.
That check was bundled into the same shell call as something unrelated: a live import against NocoBase. The NocoBase API key had expired earlier that evening, the request came back 401, and the Python on the other end of the pipe fell over:
(not reading its contents)
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
KeyError: 'data'The first line is the key-file check, which located the file and politely declined to print it. Everything after it is the NocoBase half. The assistant went off to diagnose the expired NocoBase key, which was the louder problem, confirmed it with a 401 ... Your session has expired, saved a new key, ran the RSS import, wrote the ChatOps setup script, dry-ran it with stubbed sudo, and wrote the tests. The key-file question was never picked up again.
About five minutes after that check, it committed with a blanket add. 21:40:25 local time. The file was in.
I don't think the lesson is "the assistant was careless", or not only that. I've done the same thing by hand many times: start a check, get distracted by an error from the thing next to it, forget the check existed. If a safety check shares a command with something that can fail noisily, the noisy failure wins. Run checks on their own, and read their output before doing anything else. And a check that ends with "found it, not reading it" hasn't answered anything yet. "Could this end up in git" has a direct answer, and it's one command:
git check-ignore -v kavita/OPDS_KEY || echo "NOT IGNORED"If that prints a rule, the file is safe from git add -A. If it prints NOT IGNORED, you have work to do before the next commit. At 21:35 it would have said NOT IGNORED.
The cleanup, and why I didn't rewrite history
The file was caught within the minute. The assistant ran a check that printed only the file's size and the shape of its contents (key names, values hidden), plus which commits contained it and whether the repo was visible from outside:
-rw-r--r-- 114 bytes kavita/OPDS_KEY
--- tracked now / in which commits:
519427c ChatOps v2: private ntfy over Tailscale, 20+ commands in 3 risk tiers, ...
--- repo visibility:
Not Found
--- shape of file (key names only, values hidden):
key: <hidden>
OPDS URL: <hidden>The "Not Found" was the visibility check: asked from outside, the repo doesn't exist, which is how a private repo should look to strangers. Then:
git rm --cached kavita/OPDS_KEY # stop tracking, keep the file on disk
# append ignore rules (below)
git add .gitignore # explicit path, for once
git commit -m "Stop tracking kavita/OPDS_KEY (credential file added by mistake); ignore *_KEY, *.key, service-account*.json"
git push
git ls-files | grep -E '_KEY$|\.key$|service-account' || echo "no secret files tracked"Commit 200a659, thirty-six seconds after the bad one. The .gitignore diff:
# credential files must never be committed
kavita/OPDS_KEY
**/OPDS_KEY
**/*_KEY
**/*.key
**/service-account*.jsonThe final check printed no secret files tracked, and a history query confirmed what we already knew: the file was present in two commits, the one that added it and the one that removed it.
Then came the question of the history. The standard advice is: rotate the credential, then purge it from history with git filter-repo or BFG, then force-push, then make every clone reset hard. I did the first part and decided against the rest.
My reasons, for what they're worth. The key was rotated within the hour, so what's in history is a dead string. The OPDS URL next to it reveals a hostname, which is already in DNS and in a dozen compose labels in the same repo. The repo is private, with one human collaborator plus whatever session is editing it. And a force-push on a repo the server pulls from means a git reset --hard origin/main on the server, whose working tree has untracked runtime files in it that I'd rather not be near when typing reset --hard.
The counter-argument is real, and I'll admit it bothers me. "Private" is a property of a GitHub setting, not of the data. If I ever flip that setting, or the account gets compromised, or I mirror the repo somewhere public without thinking (I do run mirror sync jobs between git hosts), a credential that has been invalid since October 2026 will be sitting in there. Dead, but there, and a scanner won't know it's dead. If the leaked thing had been the Google service account or the Cloudflare token, I would have rewritten history without discussion. For a revoked reading-app key, I took the lazy option and wrote it down so I'd remember it was a choice.
For the record, the purge I didn't run would have been:
# on a fresh mirror clone, never the working copy
git clone --mirror [email protected]:me/self-hosted.git
cd self-hosted.git
git filter-repo --path kavita/OPDS_KEY --invert-paths
git push --force --mirror
# then on every clone (laptop, server): fetch, and reset --hard origin/mainPlus, on a hosted forge, asking support to purge cached views of the old commits, since a force-push alone doesn't make them unreachable on the host's side. That last step is the one people forget, and the reason "just rewrite history" is less complete than it sounds.
Why .gitignore is not the fix
The ignore rules I added are fine, and they would stop this exact file and its obvious cousins. But look at them again: *_KEY, *.key, service-account*.json. Every one of those is a guess about what my next mistake will be called. The previous set of rules was also a set of guesses (.env, .api-key, ops/chatops.env), and the leaked file was called something none of them covered.
An ignore file is a deny list, and a deny list only blocks names you already know about. A secret named kavita-token.txt or opds.url or notes-dont-commit sails through all of the patterns above. I checked a couple of these against the current ignore file while writing this: kavita/opds.url is not ignored, and neither is anything ending in .env.bak.<timestamp>.
That last one isn't hypothetical. A couple of days later, while writing up a sops guide for my other repos, the assistant ran git check-ignore across the tree and found that one of my own helper scripts, the one that adds a GitHub notifier user to ntfy, does this before editing the environment file:
cp .env ".env.bak.$(date +%s)"A plaintext copy of the ntfy credentials, with a name no rule matches, sitting in the app folder until I delete it. The next blanket add would have committed it, exactly the way the Kavita key went in. As of writing that gap is still open; the fix (an .env.* pattern with !.env.example re-included) is in the migration plan I haven't run yet.
So the fix that actually changes outcomes is to stop staging things I haven't looked at. The rule we wrote into the assistant's notes the same evening, and which I now follow by hand too:
- Run
git status --short --untracked-files=allbefore staging anything. - Read every
??line. Each one is either intended, or gets ignored, or gets moved out of the tree. - Stage by explicit path. No
git add -A, nogit add ., nogit commit -ain this repo. - Never print a secret file's contents to check it. Names and sizes only.
In practice it looks like this:
git status --short --untracked-files=all
# M ops/chatops.py
# M ops/COMMANDS.md
# M nocobase/provision/mods_rest.py
# ?? nocobase/provision/wf_kavita.py
git add ops/chatops.py ops/COMMANDS.md nocobase/provision/mods_rest.py \
nocobase/provision/wf_kavita.py
git diff --cached --stat # one more look at what is actually staged
git commit -m "..."That listing is lifted from the session log of the commit right after the leak, when the assistant printed the untracked files as a separate block before staging. A few commits later its summary line was "Everything in that list is intended. Committing with explicit paths." That sentence is the whole procedure. It's dull, and for this job dull is what I want.
Daily cost: maybe thirty seconds per commit, mostly reading. The one place it gets annoying is a commit that adds a whole new app folder with eight files, where typing the paths feels silly. For that case git add newapp/ is fine, after git status --untracked-files=all has shown me all eight names. The rule is about looking, not about typing.
The second reason to stage by path: other people's half-done work
The leak was the dramatic reason. The more frequent one turned out to be that I am not the only thing writing to this working tree.
On the morning of the 14th, while one assistant session was building the new apps, a different session (also me, in another window, working on NocoBase) committed a rebrand. That commit happened to touch a file the first session had modified and not yet committed, and overwrote its change. The first session noticed when git status stopped showing the file as modified, and re-applied its edit before committing. If it had staged everything blindly, it would have committed the other session's version on top and the change would have quietly disappeared.
The same thing came back a day later from the other direction. Someone, again probably me in another window, left uncommitted edits in the Dozzle compose file (two extra read-only flags on the socket proxy) and an untracked NocoBase script. The note in the assistant's memory is blunt: "never git add those blindly". A blanket add would have shipped half-finished work from a different task under a commit message that described something else entirely.
And then, two days after we'd written the rule down, it happened anyway, by my own hand. On the 16th I committed a small Dashy change. The commit is called "change grafana stack" and its stat says:
dashy/conf.yml | 4 ++--
nocobase/provision/home_today.py | 2 +-
nocobase/provision/wf_prayer.py | 49 ++++++++++++++++++++++++++++++++++++----
nocobase/provision/wf_today.py | 9 ++++----
4 files changed, 53 insertions(+), 11 deletions(-)One of those four files is about Grafana links in my dashboard. The other three were an assistant session's in-progress work on a NocoBase prayer-times module, which I swept up with a commit-everything. Nothing broke. The session noticed a few hours later ("Another session's commit already swept up my wf_prayer.py and wf_today.py edits; the content is in git") and carried on. But anyone reading the history for when the prayer-times source changed will find it under a commit message about Grafana, and if those edits had been half-done rather than finished, the server would have pulled broken code on the next deploy.
I include this because it's the honest version of the story. The rule doesn't enforce itself. I wrote it, I agree with it, and I broke it on a quick commit from a terminal where muscle memory types -a. I haven't added anything that would enforce it, either. No hook refuses a commit that mixes unrelated files, and I'm not sure one could. For now the rule lives in my head and in the assistant's notes, and the history shows how reliable the first of those is.
This is also why the NocoBase provisioning scripts now only ever delete things they themselves created, but that's a story about testing against production, and it has its own post.
What I'd do differently, and what's still open
The secrets live in the repo's working tree because that's where Docker Compose looks for .env by default, and because a setup.sh that writes ./.env next to its compose file is easy to reason about. The cost is that every app folder is a place where a secret can sit next to a file I want to commit. And since the secrets exist only on the server, there's a second cost I'd been ignoring: there's no backup of them. If the VPS died, every generated password and token would be gone with it, along with the ability to read databases initialised with them.
Two of my other repos (Nextcloud and Traefik) already use sops with age keys: secrets are committed encrypted, decrypted on the server at deploy time, and a check script in CI refuses any plaintext env file. When I asked why this repo doesn't, the honest answer from the assistant was that there's no recorded decision. It just grew this way, one setup.sh at a time. There is now a written migration plan for it: bootstrap the sops plumbing on the laptop, encrypt every .env and the stray credential files in place on the server (encrypting only needs public keys), copy back only the encrypted files, and commit them by explicit path. Around fifteen minutes, on paper. I haven't run it.
When I do, the guard rail that comes with it is a script that runs as a pre-commit hook and in CI and fails if git tracks a plaintext env file or a credential-looking file:
case "$b" in
*.enc.env|*.enc|*.example|*.sample|*.template) continue ;;
.env|.env.*|*.env) err "plaintext env file is tracked: $f" ;;
.api-key*|*_KEY|*.key|service-account*.json) err "plaintext credential file is tracked: $f" ;;
esacI'll point out the obvious, since I'd want someone to point it out to me: that's another list of names. It would catch .env.bak.<timestamp>, which the current ignore file doesn't, and it would have caught OPDS_KEY. It would not catch opds.url. The real improvement sops brings is a different one: once secrets are meant to be committed encrypted, there's less reason for plaintext credential files to be lying around in the tree at all. That shrinks the problem rather than guessing at it.
I also haven't put a content-based secret scanner (gitleaks or similar) in a pre-commit hook. It would have caught a file with a long random string in it, whatever it was called. It also wouldn't help much with the assistant unless the hook is installed in the clone it commits from, which it would be, since that's the same laptop clone. It's on the list of things I keep meaning to do.
One more thing I changed, smaller but real: when I hand a credential to the assistant now, I don't drop it into the repo folder. NocoBase secrets go into NocoBase's own environment variables, server-side secrets go into the server's .env, and the one file the assistant does read locally (the NocoBase API key) has been in .gitignore since the day it was created, under two patterns. The Kavita key that leaked now lives where it should have been all along: in environment variables on the server and in NocoBase, never in a file in the tree.
The history still has it.