Swimming upstream for new packages

The premise

I maintain SQLCipher for PHP , which rebuilds PHP against the SQLCipher extension for encrypted SQLite databases. It tracks Ondrej Sury's PHP packages for Debian and Ubuntu, because every new upstream PHP version in Sury means a new PHP-SQLCipher build is due on my side.

The challenge is operational: I want to know the moment new PHP packages enter the Sury repository, so I can trigger a rebuild promptly.

The catch is that I do not necessarily have the Sury APT repository configured on any server of my own.

Installing it just to poll for changes would be a side effect on a host that has no other reason to carry it, and it would tell me about new packages only after apt update has run on that particular machine.

What I want is a small, dedicated watcher that reads the repository metadata directly, remembers what it has already seen, and tells me the moment something new appears. APT gives me apt-listchanges for changes inside packages I have installed; it does not give me a heads-up that new versions of packages I have never installed have entered a remote repository.

So gills is that watcher. It reads APT Release/InRelease and RPM repomd.xml metadata from an arbitrary repository, compares package versions and checksums against a local SQLite baseline that's keeping state, and routes events to webhooks, email, Slack, Signal or stdout.

It also can run as a one-shot command under cron or a systemd timer, and handle its own retries.

Keeping it simple

Gills uses SQLite and has just two commands:

gills -c config.yml check
gills -c config.yml prune

check runs once. Cron or a systemd timer invokes it periodically. Notifications and their retries happen automatically during checks, according to your YAML.

prune deletes old completed SQLite history while retaining baselines and pending work.

gills -c config.yml prune            # Delete completed history older than 30 days
gills -c config.yml prune --days 7

Prune removes old completed events, their delivery records and unreferenced completed batches, then compacts SQLite.

A configuration

Here's a Sury example config that motivated the project. It watches PHP 8.3 and 8.4 sources in trixie/main, excludes alpha/beta/RC versions, and waits for their matching CLI binaries on amd64 before notifying. A minimal configuration looks like this:

version: 1
state_dir: ./state
notify: true
watches:
  - name: sury-php
    type: apt
    url: https://packages.sury.org/php/
    suites: [trixie]
    kinds: [source]
    filters:
      sources: ['php8.3', 'php8.4']
      version_exclude: '(?i)(alpha|beta|rc)'
destinations: {}

All paths are relative to the YAML file. An empty destinations mapping means results only appear on stdout. Remove filters to watch all packages in the configured suites/components/architectures. Set kinds: [source, binary] to watch both types.

The first successful check quietly establishes a baseline. Set notify_initial: true on a watch to announce existing matching packages too. Subsequent checks report additions, version changes and changed checksums.

Dry run

gills -c config.yml check --dry-run

A dry run prints the check results and proposed events as JSON. An existing SQLite baseline is read through a read-only connection and copied into memory. The real database and notification queue are left untouched. A first-ever dry run does not create a state directory. Temporary metadata downloads are discarded afterwards. Notifications are skipped. The command returns after one check.

Filtering and 'readiness'

Gills gives you a lot of control over what counts as an event:

• Select suites, components, architectures and source/binary package kinds.
• Include/exclude package and source globs, regexes and native version ranges.
• version_policy: upstream triggers on upstream-version changes; version_policy: packaging selects packaging revisions; any selects both.
• Group related binary changes by source name and source version.
• Use readiness to wait for source and required binary metadata before notifying.

The version_field setting controls which part of a version the version_include / version_exclude regexes test. For 2:8.4.1-3, upstream is 8.4.1; full includes epoch and revision. Native Debian/RPM comparators handle epochs, revisions and prereleases, so 8.4.1-1 → 8.4.2-1 is an upstream change while 8.4.1-1 → 8.4.1-2 is packaging.

'Readiness' is a nuance feature that ties the Sury case together. The repository publishes a new source package first; the matching binaries might follow on their own schedule. A readiness block tells Gills to hold an event until the same source version has the required indexed binaries:

readiness:
  require_source: true
  binaries: ['php*-cli', 'php*-common']
  architectures: [amd64, arm64]
  suites: [trixie]
  timeout_seconds: 3600

Each pattern must match at least one binary from the same source name and exact source version for every required architecture/suite. all/noarch satisfy any architecture. Multi-suite gates require identical full source versions. The timeout alerts once; the event remains waiting and can become ready on a later check.

Event types

Package identity includes suite, component, kind, name, architecture and full version. All selected versions are retained in the snapshot.

• package.added: a new slot, or an older version added beside the current version.
• package.updated: a newly published version above the previous highest version.
• package.downgraded: the highest available version moves backwards.
• package.repacked: the same version has changed artifact names/checksums/sizes or source association.
• package.removed: all selected versions of a package slot disappear.
• repository.error: repeated check failures, or an immediate metadata/file-integrity failure.
• repository.recovered: a successful check after an announced failure.
• readiness.timeout: a source release remains incomplete beyond its configured timeout.

Related changes are grouped by type/source/version. New architecture changes can merge into an existing waiting event. Changes arriving after delivery can create another event; use kinds: [source] with binary readiness rules for a source-driven build pipeline.

Notifications

Of course, the whole goal of gills is to actually get told about the change. So I wanted the tool to have a lot of first-class notification support.

Copy the desired destinations from examples/destinations.yml. Supported outputs are email (SMTP), webhooks, Slack incoming webhooks, signal-cli-rest-api and stdout. Select their names in each watch with destinations: [builds, email]. An empty or omitted watch list means no notifications. Destination definitions themselves do not accept watches.

version: 1
notify: true
watches:
  - name: sury-php
    type: apt
    url: https://packages.sury.org/php/
    suites: [trixie]
    destinations: [builds, email]

destinations:
  builds:
    type: webhook
    url_env: BUILD_WEBHOOK_URL
    secret_env: BUILD_WEBHOOK_SECRET
    events: [package.added, package.updated, package.repacked]
  email:
    type: email
    host: smtp.example.org
    tls: starttls
    username_env: SMTP_USERNAME
    password_env: SMTP_PASSWORD
    from: gills@example.org
    to: [builds@example.org]

Configure credentials through the named environment variables. notify: false continues tracking repository changes and printing results, while disabling both new notification queuing and delivery. Existing retries remain paused. Set enabled: false on an individual destination to disable just that destination. Changes observed while notifications are disabled are not sent retroactively.

Each ordinary check automatically attempts due deliveries, including retries from previous checks. Failed destinations retry independently with persistent backoff: 30, 60, 120, ... seconds, capped at an hour, indefinitely. Cron or timer frequency determines when a due retry is attempted. Digests are available through digest_seconds.

Webhooks receive JSON with stable event IDs, so receivers can deduplicate build requests. Optional HMAC-SHA256 signs the exact JSON body. Delivery is at least once: a timeout after the receiver accepted a request may result in a duplicate.

Webhook payloads

Webhooks receive schema_version, delivery id, created_at, and an events array. Each event has its own stable id, watch, repository, backend, observed_at, type, and relevant change details. Package events include source name/version and full old/new package records. Captured sources carry original URLs, hashes, names, sizes and local object paths. Times are UTC Unix seconds.

HTTP headers:

• X-Gills-Delivery: stable delivery ID.
• Optional X-Gills-Signature: sha256=...: HMAC over the exact JSON bytes.

Signal

Gills has direct Signal support through signal-cli-rest-api . The destination assumes an existing linked signal-cli-rest-api service reachable to Gills:

destinations:
  signal:
    type: signal
    url_env: SIGNAL_SEND_URL            # Complete URL, e.g. http://signal:8080/v2/send
    allow_private_networks: true       # Local signal-cli service
    allow_http: true
    number_env: SIGNAL_NUMBER
    recipients: ['+61000000000']
    events: [repository.error, repository.recovered, readiness.timeout]

The example explicitly permits a local, plain-HTTP Signal service. Set both flags false for a public HTTPS service. TLS validation cannot be disabled.

Scheduling and SQLite cleanup

Cron, every five minutes:

*/5 * * * * /usr/bin/gills -c /etc/gills/config.yml check >>/var/log/gills.log 2>&1

The templates in packaging/systemd/ invoke the same one-shot check. For a native package installation, create a gills service account, put the YAML at /etc/gills/config.yml, and set state_dir: /var/lib/gills. Copy the service and timer into /etc/systemd/system, then:

sudo systemctl daemon-reload
sudo systemctl enable --now gills.timer

Docker

Docker also runs a single check and exits:

cp examples/sury.yml config.yml
# Set state_dir: /var/lib/gills in config.yml
docker compose build
docker compose run --rm gills check --dry-run
docker compose run --rm gills check
docker compose run --rm gills prune --days 30

Schedule docker compose run --rm gills check with cron or a host timer if preferred. There is no built-in scheduler, restart loop or polling daemon.

Security model

Given gills contacts remote endpoints, I wanted a good security model as well. You can read more about it here .

A Node-RED flow for the webhook

The following example Node-RED flow receives the payload via the 'webhook' notification model and then handles it, in this case, by sending it on to Signal.

Since Gills has direct Signal CLI notification support, an example that forwards a Gills webhook into Signal is, I admit, a bit contrived. But the main point is the webhook pattern itself: the HMAC-SHA256 secret validation against the raw request body, and the parsing of the event JSON. The downstream action could be anything that Node-RED can reach: a GitLab CI trigger, a Home Assistant event, a Telegram message, etc. I have used Signal here because it is the same shape as the Enroll diff webhook and shows the pattern end-to-end.

The flow starts with an HTTP input node listening for POST on a long, unguessable URL. The first stop is a function node called Check secret that verifies the X-Gills-Signature header against the raw request body using the shared secret stored in the flow's environment as GILLS_WEBHOOK_SECRET. The HTTP In node has skipBodyParsing enabled so that the function receives the raw bytes; the signature is computed over exactly those bytes, and JSON parsing happens only after authentication succeeds.

const crypto = global.get("crypto");
const secret = env.get("GILLS_WEBHOOK_SECRET");

function reject(status, message) {
    msg.statusCode = status;
    msg.headers = { "content-type": "text/plain" };
    msg.payload = message;
    return [null, msg];
}

if (!crypto || !secret) {
    node.error("Webhook verification is not configured");
    return reject(500, "Webhook verification unavailable");
}

if (!Buffer.isBuffer(msg.payload)) {
    node.error("Enable Skip body parsing on the HTTP In node");
    return reject(500, "Raw request body unavailable");
}

const signature = msg.req?.headers?.["x-gills-signature"];

if (
    typeof signature !== "string" ||
    !/^sha256=[0-9a-f]{64}$/i.test(signature)
) {
    return reject(401, "Invalid webhook signature");
}

const expected = crypto
    .createHmac("sha256", secret)
    .update(msg.payload)
    .digest();

const received = Buffer.from(signature.slice(7), "hex");

if (
    received.length !== expected.length ||
    !crypto.timingSafeEqual(received, expected)
) {
    return reject(401, "Invalid webhook signature");
}

// Parse only after authentication succeeds.
try {
    msg.payload = JSON.parse(msg.payload.toString("utf8"));
} catch {
    return reject(400, "Invalid JSON");
}

if (!msg.payload || !Array.isArray(msg.payload.events)) {
    return reject(400, "Invalid Gills payload");
}

return [msg, null];

A few details worth highlighting. The signature format is sha256=<hex>, hence the slice(7) to strip the prefix before decoding. The comparison uses crypto.timingSafeEqual on equal-length buffers, which avoids short-circuit timing leaks. The expected digest is computed over the raw msg.payload buffer (the exact bytes Gills sent), so any proxy or middleware that re-serialises the body will break the signature, which is the desired behaviour. JSON parsing and the events array shape check happen only after the signature verifies.

Once the payload is authenticated, a Parse function turns the structured event JSON into a human-readable summary. It maps each event type to an emoji-prefixed label, walks the changes array, and prints the package name with its suite/component/architecture and the version transition:

let batch;

try {
    batch = typeof msg.payload === "string"
        ? JSON.parse(msg.payload)
        : msg.payload;

    if (!batch || !Array.isArray(batch.events)) {
        throw new Error("Expected a Gills payload with an events array");
    }
} catch (err) {
    node.error(`Invalid Gills payload: ${err.message}`, msg);
    return null;
}

const labels = {
    "package.added": "🆕 Package added",
    "package.updated": "⬆️ Package updated",
    "package.downgraded": "⬇️ Package downgraded",
    "package.repacked": "📦 Package repacked",
    "package.removed": "🗑️ Package removed",
    "repository.error": "⚠️ Repository error",
    "repository.recovered": "✅ Repository recovered",
    "readiness.timeout": "⏳ Publication incomplete"
};

const clean = value =>
    String(value ?? "").replace(/[\r\n\t]+/g, " ").trim();

const sections = batch.events.map(event => {
    const lines = [
        labels[event.type] || clean(event.type),
        `Watch: ${clean(event.watch)}`
    ];

    for (const change of event.changes || []) {
        const pkg = change.new || change.old;
        if (!pkg) continue;

        const location = [
            pkg.suite,
            pkg.component,
            pkg.architecture
        ].filter(Boolean).map(clean).join(" / ");

        lines.push("", `${clean(pkg.name)} (${location})`);

        if (change.old && change.new) {
            lines.push(`From: ${clean(change.old.version)}`);
            lines.push(`To: ${clean(change.new.version)}`);
        } else {
            lines.push(`Version: ${clean(pkg.version)}`);
        }

        if (change.level) {
            lines.push(`Change: ${clean(change.level)}`);
        }
    }

    if (event.message) {
        lines.push("", clean(event.message));
    }

    return lines.join("\n");
}).filter(Boolean);

msg.payload = sections.join("\n\n");
return msg;

The summary is wrapped into the Signal REST CLI's /v2/send payload and posted to that service. Gills gets an acknowledgement as soon as the webhook receives the payload, and its own retry queue handles the rest.

The full flow is available as a sanitised JSON file with the webhook URL and phone numbers replaced by placeholders:

Download gills-nodered-flow.json

In Node-RED: hamburger menu → Import → select the file. After importing, replace /your/secret/webhook/url in the HTTP in node's URL, set alice-number and bob-number in the Signal send function, and configure the GILLS_WEBHOOK_SECRET environment variable on the flow tab.

How to install Gills

You can pip(x) install gills, install it from my APT or RPM repositories, or fetch the AppImage from the releases page.

sudo mkdir -p /usr/share/keyrings
curl -fsSL https://mig5.net/static/mig5.asc | sudo gpg --dearmor -o /usr/share/keyrings/mig5.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/mig5.gpg] https://apt.mig5.net $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/mig5.list
sudo apt update
sudo apt install gills

If Gills sounds useful to you, the source and issue tracker are on my Forgejo. I'd be glad to hear how you use it.