Introducing stef: grep that sees through SOPS, and remembers what it saw

Uncovering secrets

If you run infrastructure the way I do, a large slice of your configuration tree is SOPS -encrypted: host_vars/*.sops.yml, .env files, JSON blobs. The plaintext values are never on disk. Exactly what you want! But it also means that every ordinary text tool (like a search tool) is blind to them. grep -r password inventory/ happily reports that your secrets files contain nothing of interest, because what they actually contain is AES256-GCM ciphertext.

In fact, it's worse than that: I even get false positive matches because of coincidental gibberish in the SOPS files that match the pattern I searched for. Not helpful...

The standard workaround, of course, is to decrypt those sops files by hand:

sops -d inventory/host_vars/someserver/forgejo.sops.yml | grep password

For one file, that's fine. For a real, recursive mass search, it falls over quickly:

• your tree contains hundreds of ordinary files plus a handful of encrypted ones, and grep has no idea which is which;
• you lose ignore rules, globs, file types, context lines and colouring the moment you pipe;
• each file needs its own sops invocation, so it's slow and easy to get wrong in a loop;

One other quirk that I've always wanted: a grep match is stateless. It can tell you what matches now, but it can never tell you what used to match.

There's ways around that! So long as your files are in git, you can git blame and find the associated commit for when a line changed. Then, you delve into new hell decrypting the contents of the git commit to understand what changed in that encrypted file. Only to find that the git blame is just masking an earlier change in an earlier commit that you wanted..

So I wrote stef to close that gap.

What stef does

stef is a standalone grep-style search tool. The normal muscle memory works:

stef forgejo README.md
stef -r forgejo ~/git/mig5-devops/
stef -r 'password|token' inventory/
stef -r -C 3 'failed login' /var/log/

Directory recursion is opt-in like GNU grep (-r without following symlinks, -R following them), patterns can be regexes or fixed strings (-F), and the familiar -i, -w, -v, -A/ -B/-C, -m, glob (-g), file-type (-t), ignore-file, --json, --vimgrep, -c, -l, -q controls are all there. Exit codes follow grep convention too: 0 found, 1 not found, 2 trouble.

The difference is what happens when stef meets a SOPS file. It first narrows candidates by likely configuration formats (*.yaml, *.yml, *.json, *.env, *.ini, *.sops* - extend with --sops-glob, disable entirely with --no-sops), then byte-sniffs the candidate for the ENC[AES256_GCM marker and SOPS metadata. Both signals must be present, so a README that merely mentions ENC[AES256_GCM is not mistaken for an encrypted file.

If the file is genuinely SOPS-encrypted, stef runs the sops binary you already have and trust, as the most compatible interface there is:

sops --decrypt FILE

and pipes the decrypted stdout straight into the same search engine used for plaintext files. Decrypted content is never written to a temporary file - it exists only in the stream. The original on disk stays encrypted, and your existing SOPS key setup (GPG, age, KMS) is used exactly as-is, because stef does not reimplement any of it.

That means this "just works" across a mixed tree:

stef -r password infrastructure/

with plaintext files searched directly and encrypted files decrypted and searched transparently, all in one pass, with one binary.

Built from ripgrep's own engine, without ripgrep

The first stef prototype I built shelled out to rg. That worked, but it meant asking users to install a second search tool. Worse, it created flag conflicts - GNU grep's -r means recursive while ripgrep's -r means replacement. Later, I realised that ripgrep is actually a Rust project that splits its functionality into a sort of 'modular' series of crates that can be used independently.

So, stef is now a single native Rust binary that embeds the reusable crates from the ripgrep project itself - grep-regex, grep-matcher, grep-searcher and ignore - for the regex engine, line-oriented search, and directory traversal with .gitignore/.ignore/.stefignore support, globs and file types.

So stef is not a half-finished reimplementation of a regex engine; it searches with the same machinery ripgrep does. It is also deliberately not the full rg CLI: there is no PCRE2, no replacement output, no -z compressed-file search, no --pre preprocessors. Those omissions are intentional. In the case of preprocessing, SOPS is a first-class integration rather than something you bolt on with --pre.

The part grep can't do: remember previous matches

Every ordinary search is remembered (unless you pass --no-history). Not file copies - only the match records the search produced, plus enough metadata to replay the search. Then:

stef -p -r database_password inventory/

compares today's matches against the most recent distinct earlier result for that same search, diff-style:

@@ inventory/group_vars/all/secrets.sops.yml @@
-     12 │ database_password: alpha
+     12 │ database_password: beta

Red is what disappeared, green is what appeared. Changed values show up as a removal plus an addition, because the comparison is over decrypted line text. Line movement is not a change - identity is path + matched line text, so a secret moving from line 12 to line 97 doesn't produce a false alarm. And stef -p with no other arguments replays your last remembered search automatically, including returning to the directory where it originally ran.

There's also a time window:

stef -p 7d -r forgejo inventory/

That considers every remembered snapshot of that exact search over the last seven days and compares the union of what was seen with what exists now. If a value went alpha → beta → gamma during the week and is delta today, the old values show as removals and delta as an addition - rather than quietly vanishing because no single earlier snapshot matches "the last one" cleanly.

I'll admit, this is not yet a perfect system. The only way for matches to enter the historical search is, well, to search for them. I think there's still an uncomfortable gap there, where contents might change independently of a git commit or a previous stef search, perhaps even several times over - and that change history will still be lost. (As I write this, it occurs to me that readers might be interested in looking at my article about GuardUtils , because that sort of thing feels like it could be solved with mirro! Hey Marco, can we get mirro to 'backup' SOPS files as they get edited?)

Encrypting history

In my world, the matches themselves, if we are going to store those in state, are secrets too. A tool that persisted secret lines in a plaintext SQLite database next to your dotfiles would be a liability, not a feature. So the history store is encrypted at rest as well.

I thought about using SQLCipher, but decided we have a cheap way out: to be 'grepping' SOPS files encrypted with GPG, and therefore decrypting them, this also means we can leverage the user's GPG key for the history too. So:

• match records, paths, working directories and commands are serialised to JSON, then encrypted with ChaCha20-Poly1305 under a random 256-bit master key;
• search identities in the database are keyed BLAKE3 hashes rather than plaintext pattern columns, so even the query isn't sitting there in the clear;
• the state directory is created 0700 and files 0600, and stef refuses to use a history key whose permissions are looser;
• when you search PGP-backed SOPS files, stef reads the PGP fingerprints from the encrypted SOPS metadata without decrypting anything, checks them against your local GPG secret keyring, and if one matches, wraps the master key to it as history.key.gpg - so no plaintext history key is left on disk at all. age-only SOPS users get the local-key mode instead;
• stef --clear-history deletes and then VACUUMs the database, because a secrets database that leaves the deleted rows in freed SQLite pages is not really deleted;
• stef --history shows run counts and commands, but never stored match text.

Getting it

Install from my APT repository:

sudo mkdir -p /usr/share/keyrings
curl -fsSL https://mig5.net/static/mig5.asc | sudo gpg --dearmor -o /usr/share/keyrings/mig5.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/mig5.gpg] https://apt.mig5.net $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/mig5.list
sudo apt update
sudo apt install stef

or cargo install stef, or build from source (Rust 1.88+). Plaintext searching needs nothing else, but searching SOPS files still needs sops installed separately, and configured as you already have it (and you probably already have it, else you wouldn't have the need for stef in the first place).

The source code, as usual, is in my Forgejo .