Show me the evidence!

An organisation running their operations according to a framework (such as ISO27001, SOC-2 etc) produces evidence every day: software upgrade logs, a reviewed change, the act of someone SSHing in from a VPN with an SSH key, an Ansible playbook run, a successful restore test, an access review, a supplier update.

Making that material useful for assurance takes context. Which requirement does it support? Which service and period does it cover? Who reviewed it? What needs to improve?

It also is recorded 'as it happens', whereas frameworks tend to be structured very strictly through 'Controls' and 'Clauses'. The order of these Controls makes sense to an auditor's brain, but the reality down here in the trenches is different.

I've spent a lot of time preparing for and undergoing audits with organisations. I would assemble all the most relevant procedures and other documents which explain how infrastructure is managed. I'd collect screenshots or hyperlinks to really good examples of CI/CD jobs, Pull Requests, and other records. However, come the actual audit time, I'd find out that the auditor wants to start with a specific Control. Very quickly, we would be jumping all over the place as I scrambled to find the right thing to show.

The frustrating part has always been that I know the evidence is there. But the order in which it made sense to me to show it, was not the way the auditor would see it.

Or, I'd show a document saying what we are supposed to do, but the act of proving that's what we do do, can be trickier.

Wouldn't it be great if all the actual raw evidence was already there and already mapped appropriately to the relevant Control?

Then the auditor could simply pick the Control they're interested in, immediately see evidence, with contextual remarks explaining why that evidence is relevant to the Control (since often evidence is very 'raw' and it needs context). They could then 'sample' that evidence, ideally very safely (e.g with redactions if necessary, as well as headers/footers annotating where it has come from).

You'd also ideally want that evidence to be near 'tamper-proof', e.g in some sort of immutable storage that helps assure its provenance.

And why not also map other 'entities' to Controls, such as the documents themselves, the recognised risks, the assets in your asset tracking system?

To actually ingest that evidence and have it automatically associated with your framework is really demonstrating that you aren't just compliant: instead, your organisation is demonstrating that it lives and breathes its ISMS, day in, day out.

So, I created KEEN - the Key Evidence Engine to connect those records with frameworks, risks, audits and the Information Security Management System (ISMS). Start with an event, trace it to a control, inspect the supporting material and record your assessment.

KEEN is free and open source software . You can run it on your own infrastructure, on-premises or in the cloud, and shape its framework definitions and evidence rules around your organisation.

Where KEEN fits alongside other products

A lot of products out there are 'lip service'. You spend a lot of money on a risk management platform. But all you really get to do is tick some boxes saying 'yes, we do that'. You also pay through the nose, usually by headcount, via sales consultants under pressure to meet KPIs.

KEEN offers an open source, self-hostable approach for organisations that want to work directly with the underlying evidence and tailor the system themselves.

The starting question is: what demonstrates that this practice is operating in our organisation? In KEEN, you define how your operational events relate to your requirements, retain their context, and use them in a continuing programme of measurement, review and improvement.

That is useful when your evidence lives in bespoke systems, your business practices need specific mapping rules, or you want control over hosting and development. You choose the environment, maintain the source systems that provide the data, and decide how the assurance model evolves.

Collect from the systems you already use

Built-in ingesters in KEEN are like presets - they come with the product out of the box. These include GitHub, GitLab, Forgejo, Gitea, Jenkins, Redmine, Loki, Amazon CloudWatch Logs, Google Workspace, Taiga, BookStack and RSS/Atom feeds. You can also push data to authenticated KEEN webhooks, and add manual 'diary' evidence.

KEEN Agent is a complimentary open-source daemon which can be installed on your Linux servers. It collects things like journal syslog messages, nginx requests, package-manager activity, auditd records, and even OSSEC json logs. This is useful if you're not already shipping your logs to a central service like Loki.

For a bespoke service, there is also a API ingester builder. This lets you construct completely unique-to-you HTTP requests, credentials, pagination and event extraction in the web UI, which will then call those endpoints of your bespoke services, and attempt to ingest the response. Compatibility depends on the remote API's authentication and response format.

Redaction and masking help limit sensitive data entering evidence workflows. Administrators can pause collection and set evidence retention policies, with protection for records sampled into audits.

Map once, review the evidence in context

Mapping the evidence to controls is an act of curation - it often is spiky at the start until you climb the curve of your evidence 'patterns'.

To make it easier, you can pick an existing event and create a mapping rule from it to apply to it and all similar events both retrospectively and into the future. Keep the fields that matter, choose its framework targets, and preview matching and nonmatching records. A software-maintenance rule might select package activity; a more specific web-server rule might select a status code, path and client network.

Many organisations need to comply with multiple frameworks. KEEN includes many framework catalogues thanks in large part to work performed by others to collate and cross-reference those controls. KEEN offersr a framework editor so that you can edit or add new frameworks as you need. You can enable the frameworks your organisation needs.

For me, a killer feature that exists in KEEN is that you can associate the controls from one framework to another - what this means is that any evidence arriving and mapping to one framework's controls, automatically will map to that other framework's controls, without you needing to do anything at all!

Of course, when writing mapping rules, you can also explicitly map any number of specific controls to that evidence pattern.

Keep the ISMS connected

Most certified organisations have a wide range of documents, spreadsheets, matrices and the like in a range of different systems.

KEEN makes it possible to manage assets, people, organisational roles and charts, suppliers, access-control matrices, objectives and meeting minutes - all alongside your other ingested evidence. This means that those entities are also able to be 'related' to Controls directly. Once you start to build those relationships, you can also visualise them in interesting ways.

Write policies and procedures directly in KEEN, retain document revisions, upload existing material or link to external documents. If you are using BookStack, integration with its API can mean you can associate even specific parts of a book or a shelf to your framework.

Risk registers bring together inherent and residual assessments, 5×5 heatmaps, treatment plans and ownership by a person or organisational role. There is support for CIA (Confidentiality, Integrity and Availability) risk models, as well as PESTLE.

Not sure how to categorise your risks? The KEEN Mitigator uses a natural-language (not AI!) analysis of your description to suggest both controls that may apply as mitigation, and whether you or someone else has already defined a similar risk, to help keep your data hygiene nice and healthy.

KEEN also ships with its own informal 'KEEN Assurance Framework', as inspiration for both modelling your organisation against Controls and as a reusable 'risk library' for you to pick and choose which pre-defined risks are relevant to your business.

Measure whether the practice is effective

This is the crunch point for the auditors. It's all well and good to say you try to reach five nines, or that your data recovery procedures hope to reach a Recovery Time Objective of 20 minutes, but do they, in practice?

In KEEN you can define an 'effectiveness measurement', its unit and thresholds (be it time, percentage, a count, greater than or less than or equal to, etc), then record observationsL: either manually (with evidence!), or automatically through an inbound metric webhook from the systems that generate those metrics. KEEN gives you copyable Python and curl payload examples that KEEN expects to receive on those webhook endpoints, to help connect those reporting systems easily.

For example, measure the percentage of critical patches completed within an agreed SLA. Retain the reporting period and supporting evidence, compare the result with its target, and investigate any missed threshold.

As always, those effectiveness measurements can be directly mapped to Controls! This means the auditor, when viewing a Control, can easily find those measurements from the same page.

Audit, sample and improve

Schedule recurring audits or begin an on-demand review. Set its scope, sample evidence into the audit, document OFIs or non-conformities, and retain the assessment. Evidence can also be downloaded for review. Scope-gap views help identify areas that still need audit attention.

Auditors can questions about an event (they frequently need more context). Those and their answers are kept alongside the evidence. For more formal issues arising out of evidence, incidents can also be declared, which can integrate with your own ticket system for follow-up.

Managing use of KEEN

KEEN has its own local user management system with full RBAC for controlling who can do what, as well as support for security keys and TOTP Authenticators to help secure accounts with 2FA. An audit log also exists for actions taken in KEEN, so you can even audit the audit system!

If you don't want to use the local user auth, there is support for SSO (Google, Github, bring-your-own OpenIDConnect, or LDAP).

Work with the person who built it

Like all open source software, the onus can sometimes be on you to keep it running well. But it doesn't have to stop with you.

As KEEN's inventor and developer, I also offer professional services to help your organisation put it to work:

• Install and operate: deployment in your chosen hosting environment, configuration, upgrades, maintenance and ongoing operational support.
• Connect and adapt: ingester setup, evidence mapping, additional features and custom development for your services and workflows.
• Migrate your ISMS: I can help bring existing policies, assets, risks, ownership and evidence systems into KEEN, with a practical plan for populating and maintaining the records.
• Shape your assurance programme: If you're new to compliance, I can work through your specific framework requirements and business practices to define useful evidence, measures, mapping rules and audit routines, to help you get closer to that certification!

KEEN is a means to an end, but that end is seriously impressive: a populated, maintainable system that reflects how your organisation works - not because you say so, but because it proves it, before the auditor even has to ask.

Contact me to discuss your implementation or migration .

Try KEEN

If you need convincing, you can start a free 30-day demo at keen.cloud . Allow 5 to 6 minutes for your private, free instance to be prepared. The demo will be seeded with fictional sample records to help give a sense of the experience.

After 30 days, it will shut down and delete itself automatically. There is no obligation to continue with a paid engagement unless you want to get in touch!

You can also read the handbooks or download the illustrated brochure if you want to pass on some succinct detail to your auditor or Q/A team.

If you're 'keen', I'd love to hear from you!

Tracing a very particular voyage

A cormorant glides past and settles on a port-side cardinal mark, where the Western Port channel starts. Water slaps gently against the hull of my father's boat as I reach for the laptop, running OpenCPN - great open source nautical chart software, by the way.

I curse under my breath as my e-reader slips off my lap, its screen showing a close up image of Jean Le Cam 's keen, inquisitive face - a Breton yachtsman and competitive sailor whose name is very well known, especially in the French sailing community. He's a cheeky devil, known to speaking bluntly, but always honestly. This book of his I'm reading is a delight - actually a sort of transcribed long-form series of interviews.

A point he keeps returning to is that he is not prone to nostalgia: he always looks ahead, to what's next. With a wry laugh, he also admits he is prone to making the same mistakes incessantly.

I wish I was one of those people who can just look ahead, but I'm quite nostalgic, and I often feel very viscerally shaped by the experiences or decisions that led me to the present moment. I look backward, inwardly, all the time. Not to wistfully wish for change, but to better understand why things happened the way they did.

It's a particular time at the moment in that regard - probably to a large degree because I have reached my early 40s. But especially professionally, I am thinking about where it all began for me, and what's coming next. So where did, in fact, 'mig5 system administration' start for me?

Golden years

In 2010, I was working at a web agency that was going through a dry spell. They wanted to save money by sacking my junior sysadmin. Angered by the idea of this solution, I argued that this junior was earning one of the lowest salaries at the org, but was doing a great job. If anyone ought to leave, it, in my opinion, should have been the director of New Business - because where was it? What was he doing while my junior continued to get his work done, meeting his own and my expectations?

Le Cam probably could not have been blunter. It went down about as well as you'd expect. After a few days of arguing and generally being a pain in my boss' backside, I negotiated myself to go down to three days a week, and my junior to four. Between the two of us, we reduced roughly the equivalent of his original wage, for the organisation to stay afloat.

On those two days off per week, I started taking sysadmin consulting work on the side. It took less than a year before I was reliably earning more money on those two days a week than I was at the 'day job'. By October 2011, freelancing became my whole living, and it has stayed that way, almost by accident.

How did I find the work? Rather incredibly, it found me. From around 2009 I had become a core developer of Aegir , a site management and hosting system for Drupal, and somewhere along the way I found myself pioneering ideas that the wider industry had yet to name: using Aegir's clone features to move builds through dev, stage and production , and leaning on Drush Make to pin and reproduce an entire application from a manifest. Looking back, it was configuration management 'avant la lettre'. Puppet was still finding its feet. Ansible barely existed. Containers were a curiosity at best, and the cloud itself was only a couple of practical years old. Even Github was pretty new - I noticed the other day that I registered there in June 2009 (boy, it took me a while to get to grips with Git, coming from Subversion and CVS world!).

Because I was one of the louder evangelists for that automation workflow, work came to me easily. (I also don't think, at the younger age I was, how much being a 'white guy in tech' was already working in my favour.) I was promoting tech solutions through my website via Twitter (at a time when that corner of the internet was still fun). Agencies running Drupal and Magento needed someone who understood the whole lifecycle, and for the better part of a decade, from 2011 to 2020, that was my world. It would be dishonest if I claimed it never went to my head. It was a fine place to be: respected, in demand, and shaping how a small corner of the industry thought about deployments. People literally were writing about having 'fallen under my spell' .

Then the music stopped

There was a bill to pay for all that visibility, and I paid it in anxiety. The Drupal and Magento ecosystems went through some ugly security incidents in those years, and when you are the infrastructure person behind an agency, every emergency belongs to somebody else's client, and somehow still to you. Some difficult clients, difficult clients of clients, difficult clients of clients of clients. Problems that always seemed to beep the monitoring alarm on my phone the moment I stepped out the door with my wife on a Saturday morning. The endless chain of liability, one step removed from anything you control. I found myself lying awake doing arithmetic about risk that, in retrospect, was never really mine to carry.

Years of CBT helped, but only to a degree (honestly? Escitalopram has done more). I learned a lot, including that one of the reasons I probably do systems engineering is due to an attraction to being 'in control' as a means to overcome the anxiety. Unfortunately, it's also an oroborus, in that wielding a lot of power brings, as they say in the comics, great responsibility, and it can be a big weight on the shoulders when you are easily 'triggered' by certain conditions.

So during the COVID lockdowns I made a clean break. I stepped away from agency work entirely and went looking for organisations with a single mission I could believe in: civil society, human rights, privacy, public-interest technology. Places where the security of the systems genuinely matters because real people are affected by it. It suited me, and it still does. Strangely enough, nothing about these missions make them less stressful - in fact, many of them are even scarier than before! But I've somehow found a way to make it emotionally resonate with me at a level that cuts out that other type of fear that, even as I try to write this article, is tricky to put into words.

The switch away from my 'community' that was Drupal, had another cost though. I stopped speaking at (or even attending) conferences, stopped going to meetups, stopped writing and making videos. Perhaps COVID didn't help either, but I don't think it was much of that. I had been one of those people who explained or even introduced ideas to an industry before the word 'influencer' existed, and then I simply went quiet. I became something of a hermit, and I let my community ties lapse. In our line of work, that is close to vanishing.

What we do in the shadows

The strange thing about the hermit years is how much of value got built there. Post-Drupal, my work settled into infrastructure as code, automation, monitoring, effective incident response, security compliance, and the general craft the industry now calls DevSecOps. I learned an enormous amount.

I also did what I have always done, which is, 'build the tool I need' and release it. Enroll , which reverse-engineers existing production servers into honest, reproducible Ansible; more recently, others such as Heckle , which pulls a whole git forge into Terraform or OpenTofu (notice this trend of trying to get to 'being back in control'?); a decade of packaging SQLCipher for PHP .

I also became a core developer of OnionShare - that project, in particular led to a bunch of paid gigs still to this day, in the Tor/privacy/security space. For a time, I was the sysadmin of the onion services of Qubes OS ' package repositories through consulting to Whonix. Qubes remains my daily driver. The OnionShare opportunity has echoes of my 'Drupal journey', with word of mouth working to my advantage, but without quite the same level of visibility.

Open source gave me my career in the first place, and it has kept giving, even during the years I gave the least of myself back.

Let me digress!

Some of the work I am proudest of is still yet to be released, but is coming soon! Let me go on a brief and total tangent because I'm excited by it. Over the course of many ISO27001 audits, I gradually learned how auditors think, and became conscious of a frequent divide between 'on the ground operations' and what a compliance framework looks like on paper. Well-run orgs often tick most or even all of the boxes! but the order they go about it is frustrating for auditors, who think about risk and process information differently. This often leads to auditors asking absurd, often already-answered-but-in-a-different-way questions that lead to frustration back for those in the trenches too.

Tired of jumping in all directions to provide the evidence the auditor was seeking, I have built a compliance platform that does something the market largely skips: ingesting real raw data from logs and other sources, and automatically mapping those to relevant compliance framework controls and clauses (such as ISO27001:2022, Cyber Essentials, SOC-2 and so on) based on established pattern matching and rule definitions. Most compliance products ask you to fill in questionnaires about your controls. Mine starts at the other end. The auditor has an interface to 'snapshot' evidence, there is automatic data masking/redactions where necessary, and full ISMS and risk management tools. My ultimate goal: not needing the auditor to even ask me questions.

Several auditors have urged me not to open source that work, because they think it is a true 'competition killer' and one that should be locked away and monetised ASAP. Nothing could convince me better that making it FOSS is therefore the right move, and so it's going to be. I'm on the cusp of developing a free demo system for it. More on this to come soon!

Anyway, on to how things look for me in October 2026.

The ship that carried me

One client has been with me for fourteen of these nineteen years. That's a long time for a freelance contract. It has been quite a high stakes client with major security needs, and so working there has not been easy for me and my anxiety. But it has been, in financial terms, a keel: the steady weight under the waterline that keeps the whole boat upright. Recently, funding and cash-flow problems has made the organisation non-viable, and it is now winding up. There is no fault to assign. It is simply a ship that has reached the end of its voyage, and I am grateful beyond measure for how long it carried me.

It does mean a steep annual drop in funds for me. For the first time in my career I am out in the market looking for work, without the old privilege of it landing in my lap. That is a peculiar feeling at my stage of life. My ego was never enormous, but I would be pretending if I said I am indifferent to being respected. I am at the age where I still need to work, and recognition and reputation still matter in ways both practical and personal. The awkward truth is that my reputation faded, yet it was while my skills increased. The Drupal-era work that made me known has been largely forgotten, and the work that replaced it happened in private, away from the room where reputations are maintained.

I've started a Buy me a Coffee for those who might benefit from my open source, volunteer time and feel they can give back. Similarly, I set up the Github 'Sponsor' feature . But honestly? I don't expect to see anything there. Times are tough already for people.

Jean Le Cam's new boat

Jean Le Cam spent years racing the Vendée Globe in IMOCA vessels at the raw, punishing edge of offshore sailing, then grew disillusioned with what that kind of racing had become. For the upcoming Route du Rhum he has chosen a much slower, far more comfortable boat. He knows he is probably not going to win. He has done the hard yards, many times over, and now he wants to enjoy the sea again. It is a return to the roots of why he sails.

I read my own situation in that. The IMOCA years, the record-chasing, the solo punch-outs in the Southern Ocean: that was agency work at scale, security incidents at 3am, reputation maintenance, the tech talks. The slower boat is this: working with the organisations I believe in, building tools that solve real problems, and returning to something I have genuinely missed, which is helping shape best practices in the Dev, Sec and Ops space, in public, with the community I drifted away from. In particular, making it easier to demonstratably prove strong security safeguards in workflows and business processes, without just lip service.

I, too, would like the sailing to be good again.

Coming back into the room

This has been a proper ramble! What am I on about, anyway? I guess this essay is, in part, a reintroduction. I am Miguel Jacq. I have worked on Linux servers since 2007, consulted since 2010, and been fully remote from Melbourne for that whole time, carving out a niche working the 'night shift' for orgs mostly located in the US, the UK, and Europe. I help small and specialist teams, often with key missions that I personally happen to believe in, untangle poorly-managed Linux estates, automate change to reduce time without skimping on checks and balances, harden security all the way through the org's processes, keep monitoring useful without alert fatigue, and make recovery feasible.

I join teams that don't need a full-time sysadmin, or need mentoring of existing ones, and try to leave behind lasting changes, playbooks and documentation so the system stops depending on any one person's memory, mine included. For a number of years, I've specialised in demonstrating 'the art of the possible' for technical problems that, for any number of reasons, have felt 'too hard'.

I am once again enjoying the work for a multitude of reasons, be it my health and wellbeing, a certain middle-aged reflection, an accomplished skill set and wide range of experience. I am older, more pragmatic, and here to help. I don't need to be a 'thought leader', but I also don't want to be forgotten. It's a hard thing to write. Google 'Australian tall poppy syndrome', it maybe partly explains why. (I am also half Australian, half French, and the 'mixed identities/cultures' phemonenon is also a big area of interest and part of who defines me - perhaps a blog post for another time.)

If you knew my name from Aegir and my Drupal days, thank you for that era and helping me be successful. I hope I inspired you or helped you run your tools too. I would love to show you what I build now. More is coming soon. I am open to new engagements, and to collaboration on the open-source work, and to simply being part of the conversation again.

Le Cam will be racing in November for the love of the sea, on a boat that finally suits the man sailing it. Funnily enough, I got my own marine license just recently, following a trip to Saint-Pierre et Miquelon which helped reconnect me to the water and my ancestral roots. It's also why OpenCPN's on the laptop now. My Dad is still sailing, still has plenty to teach me in that regard. (He happens to also be a long-time Linux guy who also liked arguing with his boss. My sysadmin adventure really began with a stack of his O'Reilly books, at a time when there was no Stack Overflow, let alone AI.)

Le Cam has renovated his boat for comfort and for fun. In some ways, I've been doing the same with my business: in public, somewhat the way it used to look, but better, and for who I am now. The conditions are a bit dicey at the moment. Tech turbulence. But I'm out there, like the cormorant. Trying to be a lighthouse of logic, in the storm, for traffic seeking safe passage.

Terraformer is gone

For years, Terraformer from GoogleCloudPlatform was the go-to tool for "reverse Terraform": pointing it at existing infrastructure, having it read the live state, and generating .tf files so you could bring something you had built by hand under Infrastructure as Code. It covered dozens of providers, and for many it was the pragmatic answer to the "we already have this thing running, how do we codify it?" question.

Unfortunately, it seems that since March 2026, the repository is now read-only, and states that no further updates, security patches or support will be provided. It's dead, Jim.

Enter Heckle: a focused alternative for git forges

I had a specific scenario that required getting a Github org with many repositories, teams, members, branch protection rules etc into OpenTofu quick smart.

After a few crusty scripts that did the job, I turned my attention to another client's Gitlab.

Then I decided to get my own Forgejo under control, otherwise, whose dog food am I eating in this industry? Not my own!

In the end, I abstracted the logic, made a lot of robustness and provider-specific guardrail fixes, and the result is Heckle .

Heckle inventories a namespace on GitHub, GitLab, Gitea or Forgejo - an organisation, a GitLab group (including nested subgroups), or a personal account - maps the supported objects to the selected provider's import contracts, and emits a standalone Terraform or OpenTofu project of ordinary .tf files and local modules.

Then it steps out of the way: after adoption, Heckle is not required. You just edit and run the generated project like any other Terraform or OpenTofu code.

What makes it safe to adopt

The hard part of reverse Terraform was dealing with provider differences - when I say provider, I mean literally the Terraform/Tofu provider for those forges. Frequently, they seem to work differently to how you use them in the web UI. Either there is contradictory stuff in the APIs, or missing bits, or the developers of the TF providers simply did it a 'different' way, or haven't yet got to adding certain features yet.

In any case, here's some info about Heckle:

• It never runs apply. Heckle reads your forge, generates HCL, and rehearses the adoption path in disposable local state. It never applies anything, and it never persists that temporary state.
• Inspect before you generate. heckle inventory saves a discovery snapshot and coverage report without generating any HCL, so you can check what the token can see and what the provider covers before committing to anything. heckle coverage reads a saved report without network access.
• Rehearsed adoption. Heckle rehearses the import privately, classifies narrowly-defined, version-scoped known provider behaviour, and stops on anything unrecognised - create, delete, replace or unexpected updates halt generation rather than reaching you as a surprise.
• Guarded by default. Generated resources carry prevent_destroy guards unless you explicitly opt out, and provider versions are pinned to exact, audited releases (heckle compatibility shows what this release has audited).

That said, the same caveat Terraformer users knew applies here: Heckle helps you understand a plan; it does not guarantee one is safe. Provider behaviour, forge APIs and upstream bugs can produce unintended changes. Review every plan yourself - the decision to apply, and responsibility for its effects, remains yours. And I'll state it bluntly: Heckle is also currently alpha software. Living on the edge!

Running it

Install with pipx, apt or dnf, or grab an AppImage - see the installation docs . Credentials go through environment variables, never command line options, and are never rendered into provider configuration.

One organisation on GitHub, using the default OpenTofu:

export GITHUB_TOKEN='...'
heckle generate github --org example --out ./github-iac

A nested GitLab group on a self-hosted instance, with Terraform instead of OpenTofu:

export GITLAB_TOKEN='...'
heckle generate gitlab --group example/platform --url https://gitlab.example.org --tf terraform

Personal accounts work across all four forges with the same selectors:

export FORGEJO_API_TOKEN='...'
heckle generate forgejo --me --url https://code.example.org

And the inspect-first workflow, if you want to see coverage before generating anything:

heckle inventory gitea --org example --url https://git.example.org --out ./snapshot
heckle coverage ./snapshot
heckle generate gitea --org example --url https://git.example.org --from-inventory ./snapshot

Out of the project directory, the workflow is the ordinary one:

cd ./github-iac
tofu init
tofu plan -out=import.tfplan
tofu show import.tfplan
# Review every change before running this yourself:
tofu apply import.tfplan

From there, the .tf files and modules are yours to maintain directly, and Heckle is no longer part of the picture.

Learn more

heckle.tf has the full documentation, provider compatibility details and the adoption flow. As always, the source code is in my Forgejo at git.mig5.net/mig5/heckle .

If you give it a spin on your own forge, let me know how it goes!

Uncovering secrets

If you run infrastructure the way I do, a large slice of your configuration tree is SOPS -encrypted: host_vars/*.sops.yml, .env files, JSON blobs. The plaintext values are never on disk. Exactly what you want! But it also means that every ordinary text tool (like a search tool) is blind to them. grep -r password inventory/ happily reports that your secrets files contain nothing of interest, because what they actually contain is AES256-GCM ciphertext.

In fact, it's worse than that: I even get false positive matches because of coincidental gibberish in the SOPS files that match the pattern I searched for. Not helpful...

The standard workaround, of course, is to decrypt those sops files by hand:

sops -d inventory/host_vars/someserver/forgejo.sops.yml | grep password

For one file, that's fine. For a real, recursive mass search, it falls over quickly:

• your tree contains hundreds of ordinary files plus a handful of encrypted ones, and grep has no idea which is which;
• you lose ignore rules, globs, file types, context lines and colouring the moment you pipe;
• each file needs its own sops invocation, so it's slow and easy to get wrong in a loop;

One other quirk that I've always wanted: a grep match is stateless. It can tell you what matches now, but it can never tell you what used to match.

There's ways around that! So long as your files are in git, you can git blame and find the associated commit for when a line changed. Then, you delve into new hell decrypting the contents of the git commit to understand what changed in that encrypted file. Only to find that the git blame is just masking an earlier change in an earlier commit that you wanted..

So I wrote stef to close that gap.

What stef does

stef is a standalone grep-style search tool. The normal muscle memory works:

stef forgejo README.md
stef -r forgejo ~/git/mig5-devops/
stef -r 'password|token' inventory/
stef -r -C 3 'failed login' /var/log/

Directory recursion is opt-in like GNU grep (-r without following symlinks, -R following them), patterns can be regexes or fixed strings (-F), and the familiar -i, -w, -v, -A/ -B/-C, -m, glob (-g), file-type (-t), ignore-file, --json, --vimgrep, -c, -l, -q controls are all there. Exit codes follow grep convention too: 0 found, 1 not found, 2 trouble.

The difference is what happens when stef meets a SOPS file. It first narrows candidates by likely configuration formats (*.yaml, *.yml, *.json, *.env, *.ini, *.sops* - extend with --sops-glob, disable entirely with --no-sops), then byte-sniffs the candidate for the ENC[AES256_GCM marker and SOPS metadata. Both signals must be present, so a README that merely mentions ENC[AES256_GCM is not mistaken for an encrypted file.

If the file is genuinely SOPS-encrypted, stef runs the sops binary you already have and trust, as the most compatible interface there is:

sops --decrypt FILE

and pipes the decrypted stdout straight into the same search engine used for plaintext files. Decrypted content is never written to a temporary file - it exists only in the stream. The original on disk stays encrypted, and your existing SOPS key setup (GPG, age, KMS) is used exactly as-is, because stef does not reimplement any of it.

That means this "just works" across a mixed tree:

stef -r password infrastructure/

with plaintext files searched directly and encrypted files decrypted and searched transparently, all in one pass, with one binary.

Built from ripgrep's own engine, without ripgrep

The first stef prototype I built shelled out to rg. That worked, but it meant asking users to install a second search tool. Worse, it created flag conflicts - GNU grep's -r means recursive while ripgrep's -r means replacement. Later, I realised that ripgrep is actually a Rust project that splits its functionality into a sort of 'modular' series of crates that can be used independently.

So, stef is now a single native Rust binary that embeds the reusable crates from the ripgrep project itself - grep-regex, grep-matcher, grep-searcher and ignore - for the regex engine, line-oriented search, and directory traversal with .gitignore/.ignore/.stefignore support, globs and file types.

So stef is not a half-finished reimplementation of a regex engine; it searches with the same machinery ripgrep does. It is also deliberately not the full rg CLI: there is no PCRE2, no replacement output, no -z compressed-file search, no --pre preprocessors. Those omissions are intentional. In the case of preprocessing, SOPS is a first-class integration rather than something you bolt on with --pre.

The part grep can't do: remember previous matches

Every ordinary search is remembered (unless you pass --no-history). Not file copies - only the match records the search produced, plus enough metadata to replay the search. Then:

stef -p -r database_password inventory/

compares today's matches against the most recent distinct earlier result for that same search, diff-style:

@@ inventory/group_vars/all/secrets.sops.yml @@
-     12 │ database_password: alpha
+     12 │ database_password: beta

Red is what disappeared, green is what appeared. Changed values show up as a removal plus an addition, because the comparison is over decrypted line text. Line movement is not a change - identity is path + matched line text, so a secret moving from line 12 to line 97 doesn't produce a false alarm. And stef -p with no other arguments replays your last remembered search automatically, including returning to the directory where it originally ran.

There's also a time window:

stef -p 7d -r forgejo inventory/

That considers every remembered snapshot of that exact search over the last seven days and compares the union of what was seen with what exists now. If a value went alpha → beta → gamma during the week and is delta today, the old values show as removals and delta as an addition - rather than quietly vanishing because no single earlier snapshot matches "the last one" cleanly.

I'll admit, this is not yet a perfect system. The only way for matches to enter the historical search is, well, to search for them. I think there's still an uncomfortable gap there, where contents might change independently of a git commit or a previous stef search, perhaps even several times over - and that change history will still be lost. (As I write this, it occurs to me that readers might be interested in looking at my article about GuardUtils , because that sort of thing feels like it could be solved with mirro! Hey Marco, can we get mirro to 'backup' SOPS files as they get edited?)

Encrypting history

In my world, the matches themselves, if we are going to store those in state, are secrets too. A tool that persisted secret lines in a plaintext SQLite database next to your dotfiles would be a liability, not a feature. So the history store is encrypted at rest as well.

I thought about using SQLCipher, but decided we have a cheap way out: to be 'grepping' SOPS files encrypted with GPG, and therefore decrypting them, this also means we can leverage the user's GPG key for the history too. So:

• match records, paths, working directories and commands are serialised to JSON, then encrypted with ChaCha20-Poly1305 under a random 256-bit master key;
• search identities in the database are keyed BLAKE3 hashes rather than plaintext pattern columns, so even the query isn't sitting there in the clear;
• the state directory is created 0700 and files 0600, and stef refuses to use a history key whose permissions are looser;
• when you search PGP-backed SOPS files, stef reads the PGP fingerprints from the encrypted SOPS metadata without decrypting anything, checks them against your local GPG secret keyring, and if one matches, wraps the master key to it as history.key.gpg - so no plaintext history key is left on disk at all. age-only SOPS users get the local-key mode instead;
• stef --clear-history deletes and then VACUUMs the database, because a secrets database that leaves the deleted rows in freed SQLite pages is not really deleted;
• stef --history shows run counts and commands, but never stored match text.

Getting it

Install from my APT repository:

sudo mkdir -p /usr/share/keyrings
curl -fsSL https://mig5.net/static/mig5.asc | sudo gpg --dearmor -o /usr/share/keyrings/mig5.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/mig5.gpg] https://apt.mig5.net $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/mig5.list
sudo apt update
sudo apt install stef

or cargo install stef, or build from source (Rust 1.88+). Plaintext searching needs nothing else, but searching SOPS files still needs sops installed separately, and configured as you already have it (and you probably already have it, else you wouldn't have the need for stef in the first place).

The source code, as usual, is in my Forgejo .

Squeezing performance out of a hotspot: database diffing

I recently released DumpTales , which compares two database snapshots offline.

But before it can compare anything, it has to read. And conventional mysqldump output is not kind to readers: one INSERT statement typically carries hundreds or thousands of rows, and a single statement can legally be 128 MiB of comma-separated tuples:

INSERT INTO `accounts` VALUES (1,'Acme','basic'),(2,'Harbour','basic'),(3,'Old Co','trial');

Splitting that into rows and values sounds trivial, until you remember that ,, ), ' and " are all perfectly legal inside a quoted value. You need a little state machine that tracks whether you are inside quotes, handles backslash escapes and doubled quotes, counts nested parentheses, and slices out values only at top-level commas. Do it wrong, and a value containing the string `'),(`` will quietly corrupt the comparison.

Dumptales is, like most of my tools, a Python app, and so I first wrote that state machine in Python, as two functions called parens and split_top.

Then I profiled, and almost all of the MySQL parsing time was sitting in those two loops. Not because the algorithm was clever or expensive - but because it is a per-character loop, and a per-character loop in an interpreter pays rent on every character.

In the end, with that same algorithm converted into a CPython extension module, the results were pretty dramatic. Nothing cleverer, just the same logic compiled to machine code.

If you're like me, you very rarely delve into C these days, but it can be kind of interesting.

And it's only 82 lines! Let's go through it.

Chunk 1: who's calling, and where are we?

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject *next_row(PyObject *self, PyObject *args) {
    PyObject *sql;
    Py_ssize_t at;
    if (!PyArg_ParseTuple(args, "On", &sql, &at)) return NULL;
    if (!PyUnicode_Check(sql)) {
        PyErr_SetString(PyExc_TypeError, "SQL statement must be Unicode");
        return NULL;
    }
    if (PyUnicode_READY(sql) < 0) return NULL;
    Py_ssize_t length = PyUnicode_GET_LENGTH(sql);
    int kind = PyUnicode_KIND(sql);
    void *data = PyUnicode_DATA(sql);
    if (at < 0 || at >= length || PyUnicode_READ(kind, data, at) != '(') {
        PyErr_SetString(PyExc_ValueError, "expected opening parenthesis");
        return NULL;
    }

The first two lines are housekeeping: Python.h is just the header that lets a C function participate in Python's world of objects and reference counting.

next_row is the one function Python will call. It receives its arguments as a tuple, and PyArg_ParseTuple(args, "On", &sql, &at) unpacks them: O means "a Python object" (the SQL statement string) and n means "a Python integer, stored as a native Py_ssize_t" (the position to start at). If the caller passes nonsense, we return NULL - in the CPython C API, returning NULL from a function is how you say "an exception has been raised".

Two sanity checks follow: the object really is a string, and PyUnicode_READY normalises the string's internal storage so we can read it safely.

The KIND/DATA/READ trio is the interesting bit. Python does not always store strings the same way internally: a string of plain ASCII may use one byte per character, while a string with an emoji in it uses four. Rather than caring, we ask the string for its kind (1, 2 or 4 bytes) and a pointer to its data buffer, and PyUnicode_READ(kind, data, i) fetches character i correctly for whichever shape it is. This is a direct array lookup into memory the string already owns - we never copy the statement.

Finally, we check the caller has actually aimed us at an opening parenthesis.

Chunk 2: four bits of state

    PyObject *values = PyList_New(0);
    if (!values) return NULL;
    Py_ssize_t start = at + 1;
    Py_UCS4 quote = 0;
    int depth = 1;
    for (Py_ssize_t i = start; i < length; i++) {
        Py_UCS4 ch = PyUnicode_READ(kind, data, i);

The whole scanner needs surprisingly little state:

• values - an empty Python list to collect the row's values into;
• start - where the value we are currently inside begins (just after the opening ();
• quote - which quote character opened the current quoted string, or 0 for "not inside a string";
• depth - how many nested parentheses we are inside. It starts at 1, because we are already inside the tuple's own (.

Then the loop: walk every character from start to the end of the statement. Everything below happens inside this loop.

Chunk 3: inside quotes

        if (quote) {
            if (ch == '\\') {
                if (i + 1 < length) i++;
                continue;
            }
            if (ch == quote) {
                if (i + 1 < length && PyUnicode_READ(kind, data, i + 1) == quote) {
                    i++;
                    continue;
                }
                quote = 0;
            }
            continue;
        }

If we are inside a quoted string, the only characters that matter are:

• a backslash, which escapes the next character - so \' inside a single-quoted value does not end the string. We skip the next character by bumping the loop counter manually;
• a quote character matching the one that opened the string. If it is immediately followed by another identical quote, that is the SQL doubling convention ('' meaning one literal quote), so we skip both. Otherwise the string has ended, and quote goes back to 0.

Everything else inside a string is ignored. That is the entire escaping story: get this right and a value like `'),(`` is just data.

Chunk 4: structure, boundaries, and finding a way out

        if (ch == '\'' || ch == '"' || ch == '`') {
            quote = ch;
        } else if (ch == '(') {
            depth++;
        } else if (ch == ')') {
            depth--;
            if (depth == 0) {
                Py_ssize_t left = start, right = i;
                while (left < right && Py_UNICODE_ISSPACE(PyUnicode_READ(kind, data, left))) left++;
                while (right > left && Py_UNICODE_ISSPACE(PyUnicode_READ(kind, data, right - 1))) right--;
                PyObject *token = PyUnicode_Substring(sql, left, right);
                if (!token || PyList_Append(values, token) < 0) { Py_XDECREF(token); Py_DECREF(values); return NULL; }
                Py_DECREF(token);
                PyObject *result = PyTuple_New(2);
                if (!result) { Py_DECREF(values); return NULL; }
                PyObject *position = PyLong_FromSsize_t(i + 1);
                if (!position) { Py_DECREF(result); Py_DECREF(values); return NULL; }
                PyTuple_SET_ITEM(result, 0, values);
                PyTuple_SET_ITEM(result, 1, position);
                return result;
            }
        } else if (ch == ',' && depth == 1) {
            Py_ssize_t left = start, right = i;
            while (left < right && Py_UNICODE_ISSPACE(PyUnicode_READ(kind, data, left))) left++;
            while (right > left && Py_UNICODE_ISSPACE(PyUnicode_READ(kind, data, right - 1))) right--;
            PyObject *token = PyUnicode_Substring(sql, left, right);
            if (!token || PyList_Append(values, token) < 0) { Py_XDECREF(token); Py_DECREF(values); return NULL; }
            Py_DECREF(token);
            start = i + 1;
        }
    }
    Py_DECREF(values);
    PyErr_SetString(PyExc_ValueError, "unbalanced parentheses");
    return NULL;
}

Outside quotes, only four characters matter:

• ', " or ` open a quoted string, and we remember which one;
• ( deepens the nesting (values can contain parenthesised expressions);
• ) closes one level. If that brings depth back to 0, the tuple is finished: we trim surrounding whitespace off the final value, cut it out of the statement with PyUnicode_Substring, append it to the list, and return a two-item tuple of (list_of_values, position_just_after_the_closing_paren). That position is how the caller finds the next row without being handed (or copying) the rest of the statement;
• , at depth == 1 - a comma inside this tuple but not inside a nested one - is a value boundary. Same ritual: trim, slice, append, and remember that the next value starts after the comma.

The Py_DECREF calls sprinkled through the error paths are reference-counting bookkeeping - the C-side equivalent of Python's garbage collector, telling CPython "I am done with this object", so nothing leaks.

If the loop reaches the end of the statement without ever closing the tuple, we tidy up, raise a Python ValueError("unbalanced parentheses") and return NULL. DumpTales' Python caller catches that ValueError and re-raises it as its own DumpError.

Chunk 5: the glue that makes it importable

static PyMethodDef methods[] = {
    {"next_row", next_row, METH_VARARGS, "Parse the next SQL VALUES tuple without copying the remaining statement."},
    {NULL, NULL, 0, NULL}
};
static struct PyModuleDef module = {PyModuleDef_HEAD_INIT, "_dumptales_native", NULL, -1, methods};
PyMODINIT_FUNC PyInit__dumptales_native(void) { return PyModule_Create(&module); }

The last five lines are pure ceremony, but they are what make the whole thing work: a table registering next_row under its Python name, a module definition calling the module _dumptales_native, and the initialisation function CPython runs when you import _dumptales_native. After that, from Python's point of view, it is an ordinary module.

Why the C wins

The Python functions parens and split_top in dumptales.py implement exactly this state machine (I've kept them there in case, for whatever reason, the user doesn't or can't use the C parser).

But three things make the C version faster:

1. No interpreter tax. In the Python loop, each character costs bytecode dispatch, object handling and reference counting before the actual comparison happens. The compiled loop is a handful of machine instructions per character.
2. No intermediate copies. The Python path slices the entire tuple text out of the statement first (raw, pos = parens(s, pos)), then slices that again per value with split_top. The C version reads characters in place and creates exactly one new string per value - the minimum work needed to return Python objects at all.
3. No copying of the remaining statement. The function hands back an integer offset and the caller resumes there. The Python approach re-slices the still-unparsed tail of a statement that can be megabytes long, once per row.

Let's do some measurements!

Consider this code (you'll need to have a copy of the dumptales codebase handy and relative to the script):

import sys, time
sys.path.insert(0, '/path/to/dumptales/source')
from dumptales import parens, split_top
import _dumptales_native as native

row = "(1,'Acme & Co',45.5,'2026-09-30',`some_long_text`,NULL,'plain')"
s = "INSERT INTO `t` VALUES " + ",".join([row] * 50000)
start = s.index('(')

def python_loop():
    pos = start
    n = 0
    while pos < len(s):
        raw, pos = parens(s, pos)
        n += len(split_top(raw))
        while pos < len(s) and s[pos].isspace():
            pos += 1
        if pos == len(s):
            break
        pos += 1
        while pos < len(s) and s[pos].isspace():
            pos += 1
    return n

def native_loop():
    pos = start
    n = 0
    while pos < len(s):
        raw_values, pos = native.next_row(s, pos)
        n += len(raw_values)
        while pos < len(s) and s[pos].isspace():
            pos += 1
        if pos == len(s):
            break
        pos += 1
        while pos < len(s) and s[pos].isspace():
            pos += 1
    return n

assert python_loop() == native_loop()

for name, fn in (('python', python_loop), ('native', native_loop)):
    t = time.perf_counter()
    fn()
    print(f'{name}: {time.perf_counter() - t:.3f}s')

One INSERT statement, 50,000 rows, both parsers walking it to completion:

About eighteen times faster, consistently across runs. Multiply by a dump with hundreds of statements, and the saving is far from subtle!

In fact, real database dumps would likely stress-test the Python interpreter version even more, because wider rows, with more real string content, would increase the tax on the interpreter.

For example, if we test with wider rows:

import sys, time
sys.path.insert(0, '/path/to/dumptales/source')
from dumptales import parens, split_top
import _dumptales_native as native

def make(nrows, ncols, textlen):
    vals = ','.join(["'x%d_%s'" % (j, 'y'*(textlen//6)) if j % 3 == 1 else str(j) for j in range(ncols)])
    row = '(' + vals + ')'
    return "INSERT INTO `t` VALUES " + ','.join([row]*nrows)

def loop(s, start, native_mode):
    pos = start; n = 0
    while pos < len(s):
        if native_mode:                                                                                                                                                                            raw_values, pos = native.next_row(s, pos); n += len(raw_values)
        else:
            raw, pos = parens(s, pos); n += len(split_top(raw))
        while pos < len(s) and s[pos].isspace(): pos += 1
            if pos == len(s): break
            pos += 1
            while pos < len(s) and s[pos].isspace(): pos += 1
    return n

for label, ncols, textlen in (('narrow (7 cols, ~30 ch)', 7, 30), ('wide (20 cols, ~240 ch)', 20, 240)):
    s = make(50000, ncols, textlen)
    start = s.index('(')
    assert loop(s, start, False) == loop(s, start, True)
    out = []
    for mode in (False, True):
        t = time.perf_counter(); loop(s, start, mode)
        out.append(time.perf_counter() - t)
    print(f'{label}: statement {len(s)/1e6:.1f} MB | python {out[0]:.2f}s | native {out[1]:.3f}s | ratio {out[0]/out[1]:.0f}x')

python3 /tmp/bench_wide.py && python3 /tmp/bench_wide.py

narrow (7 cols, ~30 ch): statement 1.7 MB | python 0.38s | native 0.028s | ratio 14x
wide (20 cols, ~240 ch): statement 18.0 MB | python 3.83s | native 0.177s | ratio 22x
narrow (7 cols, ~30 ch): statement 1.7 MB | python 0.62s | native 0.038s | ratio 16x
wide (20 cols, ~240 ch): statement 18.0 MB | python 4.47s | native 0.059s | ratio 75x

So there you go. I'm pretty happy with the performance - still maybe could be better. Maybe Python and C is the wrong solution here - will I end up using Rust for the first time? We'll see :)

The premise

I maintain SQLCipher for PHP , which rebuilds PHP against the SQLCipher extension for encrypted SQLite databases. It tracks Ondrej Sury's PHP packages for Debian and Ubuntu, because every new upstream PHP version in Sury means a new PHP-SQLCipher build is due on my side.

The challenge is operational: I want to know the moment new PHP packages enter the Sury repository, so I can trigger a rebuild promptly.

The catch is that I do not necessarily have the Sury APT repository configured on any server of my own.

Installing it just to poll for changes would be a side effect on a host that has no other reason to carry it, and it would tell me about new packages only after apt update has run on that particular machine.

What I want is a small, dedicated watcher that reads the repository metadata directly, remembers what it has already seen, and tells me the moment something new appears. APT gives me apt-listchanges for changes inside packages I have installed; it does not give me a heads-up that new versions of packages I have never installed have entered a remote repository.

So gills is that watcher. It reads APT Release/InRelease and RPM repomd.xml metadata from an arbitrary repository, compares package versions and checksums against a local SQLite baseline that's keeping state, and routes events to webhooks, email, Slack, Signal or stdout.

It also can run as a one-shot command under cron or a systemd timer, and handle its own retries.

Keeping it simple

Gills uses SQLite and has just two commands:

gills -c config.yml check
gills -c config.yml prune

check runs once. Cron or a systemd timer invokes it periodically. Notifications and their retries happen automatically during checks, according to your YAML.

prune deletes old completed SQLite history while retaining baselines and pending work.

gills -c config.yml prune            # Delete completed history older than 30 days
gills -c config.yml prune --days 7

Prune removes old completed events, their delivery records and unreferenced completed batches, then compacts SQLite.

A configuration

Here's a Sury example config that motivated the project. It watches PHP 8.3 and 8.4 sources in trixie/main, excludes alpha/beta/RC versions, and waits for their matching CLI binaries on amd64 before notifying. A minimal configuration looks like this:

version: 1
state_dir: ./state
notify: true
watches:
  - name: sury-php
    type: apt
    url: https://packages.sury.org/php/
    suites: [trixie]
    kinds: [source]
    filters:
      sources: ['php8.3', 'php8.4']
      version_exclude: '(?i)(alpha|beta|rc)'
destinations: {}

All paths are relative to the YAML file. An empty destinations mapping means results only appear on stdout. Remove filters to watch all packages in the configured suites/components/architectures. Set kinds: [source, binary] to watch both types.

The first successful check quietly establishes a baseline. Set notify_initial: true on a watch to announce existing matching packages too. Subsequent checks report additions, version changes and changed checksums.

Dry run

gills -c config.yml check --dry-run

A dry run prints the check results and proposed events as JSON. An existing SQLite baseline is read through a read-only connection and copied into memory. The real database and notification queue are left untouched. A first-ever dry run does not create a state directory. Temporary metadata downloads are discarded afterwards. Notifications are skipped. The command returns after one check.

Filtering and 'readiness'

Gills gives you a lot of control over what counts as an event:

• Select suites, components, architectures and source/binary package kinds.
• Include/exclude package and source globs, regexes and native version ranges.
• version_policy: upstream triggers on upstream-version changes; version_policy: packaging selects packaging revisions; any selects both.
• Group related binary changes by source name and source version.
• Use readiness to wait for source and required binary metadata before notifying.

The version_field setting controls which part of a version the version_include / version_exclude regexes test. For 2:8.4.1-3, upstream is 8.4.1; full includes epoch and revision. Native Debian/RPM comparators handle epochs, revisions and prereleases, so 8.4.1-1 → 8.4.2-1 is an upstream change while 8.4.1-1 → 8.4.1-2 is packaging.

'Readiness' is a nuance feature that ties the Sury case together. The repository publishes a new source package first; the matching binaries might follow on their own schedule. A readiness block tells Gills to hold an event until the same source version has the required indexed binaries:

readiness:
  require_source: true
  binaries: ['php*-cli', 'php*-common']
  architectures: [amd64, arm64]
  suites: [trixie]
  timeout_seconds: 3600

Each pattern must match at least one binary from the same source name and exact source version for every required architecture/suite. all/noarch satisfy any architecture. Multi-suite gates require identical full source versions. The timeout alerts once; the event remains waiting and can become ready on a later check.

Event types

Package identity includes suite, component, kind, name, architecture and full version. All selected versions are retained in the snapshot.

• package.added: a new slot, or an older version added beside the current version.
• package.updated: a newly published version above the previous highest version.
• package.downgraded: the highest available version moves backwards.
• package.repacked: the same version has changed artifact names/checksums/sizes or source association.
• package.removed: all selected versions of a package slot disappear.
• repository.error: repeated check failures, or an immediate metadata/file-integrity failure.
• repository.recovered: a successful check after an announced failure.
• readiness.timeout: a source release remains incomplete beyond its configured timeout.

Related changes are grouped by type/source/version. New architecture changes can merge into an existing waiting event. Changes arriving after delivery can create another event; use kinds: [source] with binary readiness rules for a source-driven build pipeline.

Notifications

Of course, the whole goal of gills is to actually get told about the change. So I wanted the tool to have a lot of first-class notification support.

Copy the desired destinations from examples/destinations.yml. Supported outputs are email (SMTP), webhooks, Slack incoming webhooks, signal-cli-rest-api and stdout. Select their names in each watch with destinations: [builds, email]. An empty or omitted watch list means no notifications. Destination definitions themselves do not accept watches.

version: 1
notify: true
watches:
  - name: sury-php
    type: apt
    url: https://packages.sury.org/php/
    suites: [trixie]
    destinations: [builds, email]

destinations:
  builds:
    type: webhook
    url_env: BUILD_WEBHOOK_URL
    secret_env: BUILD_WEBHOOK_SECRET
    events: [package.added, package.updated, package.repacked]
  email:
    type: email
    host: smtp.example.org
    tls: starttls
    username_env: SMTP_USERNAME
    password_env: SMTP_PASSWORD
    from: gills@example.org
    to: [builds@example.org]

Configure credentials through the named environment variables. notify: false continues tracking repository changes and printing results, while disabling both new notification queuing and delivery. Existing retries remain paused. Set enabled: false on an individual destination to disable just that destination. Changes observed while notifications are disabled are not sent retroactively.

Each ordinary check automatically attempts due deliveries, including retries from previous checks. Failed destinations retry independently with persistent backoff: 30, 60, 120, ... seconds, capped at an hour, indefinitely. Cron or timer frequency determines when a due retry is attempted. Digests are available through digest_seconds.

Webhooks receive JSON with stable event IDs, so receivers can deduplicate build requests. Optional HMAC-SHA256 signs the exact JSON body. Delivery is at least once: a timeout after the receiver accepted a request may result in a duplicate.

Webhook payloads

Webhooks receive schema_version, delivery id, created_at, and an events array. Each event has its own stable id, watch, repository, backend, observed_at, type, and relevant change details. Package events include source name/version and full old/new package records. Captured sources carry original URLs, hashes, names, sizes and local object paths. Times are UTC Unix seconds.

HTTP headers:

• X-Gills-Delivery: stable delivery ID.
• Optional X-Gills-Signature: sha256=...: HMAC over the exact JSON bytes.

Signal

Gills has direct Signal support through signal-cli-rest-api . The destination assumes an existing linked signal-cli-rest-api service reachable to Gills:

destinations:
  signal:
    type: signal
    url_env: SIGNAL_SEND_URL            # Complete URL, e.g. http://signal:8080/v2/send
    allow_private_networks: true       # Local signal-cli service
    allow_http: true
    number_env: SIGNAL_NUMBER
    recipients: ['+61000000000']
    events: [repository.error, repository.recovered, readiness.timeout]

The example explicitly permits a local, plain-HTTP Signal service. Set both flags false for a public HTTPS service. TLS validation cannot be disabled.

Scheduling and SQLite cleanup

Cron, every five minutes:

*/5 * * * * /usr/bin/gills -c /etc/gills/config.yml check >>/var/log/gills.log 2>&1

The templates in packaging/systemd/ invoke the same one-shot check. For a native package installation, create a gills service account, put the YAML at /etc/gills/config.yml, and set state_dir: /var/lib/gills. Copy the service and timer into /etc/systemd/system, then:

sudo systemctl daemon-reload
sudo systemctl enable --now gills.timer

Docker

Docker also runs a single check and exits:

cp examples/sury.yml config.yml
# Set state_dir: /var/lib/gills in config.yml
docker compose build
docker compose run --rm gills check --dry-run
docker compose run --rm gills check
docker compose run --rm gills prune --days 30

Schedule docker compose run --rm gills check with cron or a host timer if preferred. There is no built-in scheduler, restart loop or polling daemon.

Security model

Given gills contacts remote endpoints, I wanted a good security model as well. You can read more about it here .

A Node-RED flow for the webhook

The following example Node-RED flow receives the payload via the 'webhook' notification model and then handles it, in this case, by sending it on to Signal.

Since Gills has direct Signal CLI notification support, an example that forwards a Gills webhook into Signal is, I admit, a bit contrived. But the main point is the webhook pattern itself: the HMAC-SHA256 secret validation against the raw request body, and the parsing of the event JSON. The downstream action could be anything that Node-RED can reach: a GitLab CI trigger, a Home Assistant event, a Telegram message, etc. I have used Signal here because it is the same shape as the Enroll diff webhook and shows the pattern end-to-end.

The flow starts with an HTTP input node listening for POST on a long, unguessable URL. The first stop is a function node called Check secret that verifies the X-Gills-Signature header against the raw request body using the shared secret stored in the flow's environment as GILLS_WEBHOOK_SECRET. The HTTP In node has skipBodyParsing enabled so that the function receives the raw bytes; the signature is computed over exactly those bytes, and JSON parsing happens only after authentication succeeds.

const crypto = global.get("crypto");
const secret = env.get("GILLS_WEBHOOK_SECRET");

function reject(status, message) {
    msg.statusCode = status;
    msg.headers = { "content-type": "text/plain" };
    msg.payload = message;
    return [null, msg];
}

if (!crypto || !secret) {
    node.error("Webhook verification is not configured");
    return reject(500, "Webhook verification unavailable");
}

if (!Buffer.isBuffer(msg.payload)) {
    node.error("Enable Skip body parsing on the HTTP In node");
    return reject(500, "Raw request body unavailable");
}

const signature = msg.req?.headers?.["x-gills-signature"];

if (
    typeof signature !== "string" ||
    !/^sha256=[0-9a-f]{64}$/i.test(signature)
) {
    return reject(401, "Invalid webhook signature");
}

const expected = crypto
    .createHmac("sha256", secret)
    .update(msg.payload)
    .digest();

const received = Buffer.from(signature.slice(7), "hex");

if (
    received.length !== expected.length ||
    !crypto.timingSafeEqual(received, expected)
) {
    return reject(401, "Invalid webhook signature");
}

// Parse only after authentication succeeds.
try {
    msg.payload = JSON.parse(msg.payload.toString("utf8"));
} catch {
    return reject(400, "Invalid JSON");
}

if (!msg.payload || !Array.isArray(msg.payload.events)) {
    return reject(400, "Invalid Gills payload");
}

return [msg, null];

A few details worth highlighting. The signature format is sha256=<hex>, hence the slice(7) to strip the prefix before decoding. The comparison uses crypto.timingSafeEqual on equal-length buffers, which avoids short-circuit timing leaks. The expected digest is computed over the raw msg.payload buffer (the exact bytes Gills sent), so any proxy or middleware that re-serialises the body will break the signature, which is the desired behaviour. JSON parsing and the events array shape check happen only after the signature verifies.

Once the payload is authenticated, a Parse function turns the structured event JSON into a human-readable summary. It maps each event type to an emoji-prefixed label, walks the changes array, and prints the package name with its suite/component/architecture and the version transition:

let batch;

try {
    batch = typeof msg.payload === "string"
        ? JSON.parse(msg.payload)
        : msg.payload;

    if (!batch || !Array.isArray(batch.events)) {
        throw new Error("Expected a Gills payload with an events array");
    }
} catch (err) {
    node.error(`Invalid Gills payload: ${err.message}`, msg);
    return null;
}

const labels = {
    "package.added": "🆕 Package added",
    "package.updated": "⬆️ Package updated",
    "package.downgraded": "⬇️ Package downgraded",
    "package.repacked": "📦 Package repacked",
    "package.removed": "🗑️ Package removed",
    "repository.error": "⚠️ Repository error",
    "repository.recovered": "✅ Repository recovered",
    "readiness.timeout": "⏳ Publication incomplete"
};

const clean = value =>
    String(value ?? "").replace(/[\r\n\t]+/g, " ").trim();

const sections = batch.events.map(event => {
    const lines = [
        labels[event.type] || clean(event.type),
        `Watch: ${clean(event.watch)}`
    ];

    for (const change of event.changes || []) {
        const pkg = change.new || change.old;
        if (!pkg) continue;

        const location = [
            pkg.suite,
            pkg.component,
            pkg.architecture
        ].filter(Boolean).map(clean).join(" / ");

        lines.push("", `${clean(pkg.name)} (${location})`);

        if (change.old && change.new) {
            lines.push(`From: ${clean(change.old.version)}`);
            lines.push(`To: ${clean(change.new.version)}`);
        } else {
            lines.push(`Version: ${clean(pkg.version)}`);
        }

        if (change.level) {
            lines.push(`Change: ${clean(change.level)}`);
        }
    }

    if (event.message) {
        lines.push("", clean(event.message));
    }

    return lines.join("\n");
}).filter(Boolean);

msg.payload = sections.join("\n\n");
return msg;

The summary is wrapped into the Signal REST CLI's /v2/send payload and posted to that service. Gills gets an acknowledgement as soon as the webhook receives the payload, and its own retry queue handles the rest.

The full flow is available as a sanitised JSON file with the webhook URL and phone numbers replaced by placeholders:

Download gills-nodered-flow.json

In Node-RED: hamburger menu → Import → select the file. After importing, replace /your/secret/webhook/url in the HTTP in node's URL, set alice-number and bob-number in the Signal send function, and configure the GILLS_WEBHOOK_SECRET environment variable on the flow tab.

How to install Gills

You can pip(x) install gills, install it from my APT or RPM repositories, or fetch the AppImage from the releases page.

sudo mkdir -p /usr/share/keyrings
curl -fsSL https://mig5.net/static/mig5.asc | sudo gpg --dearmor -o /usr/share/keyrings/mig5.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/mig5.gpg] https://apt.mig5.net $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/mig5.list
sudo apt update
sudo apt install gills

If Gills sounds useful to you, the source and issue tracker are on my Forgejo. I'd be glad to hear how you use it.