A security testing lab on my laptop

Testing the hell out of KEEN

KEEN has accumulated quite a lot of functionality: evidence collection, framework mappings, risks, audits, documents, questions, SWOT analysis, and all the permissions that connect those things together.

It also has a substantial test suite. Every change needs tests, and those tests need to keep up with the application. The problem is, most of those tests are functional tests, making sure things still work the way we want.

The art to security/penetration is actually the opposite: what can we throw at the app and make it do things it's not meant to do?

I wanted another way to exercise the whole installation as someone using it would encounter it: through HTTPS, the reverse proxy, a real session, the API and the browser.

There are some particularly interesting questions at that level. Can a user with one permission see a record through a different route? Does hostile text stay harmless when a page actually renders it? Does a logout invalidate the session? What happens when two requests try to consume the same one-time credential? Can you 'jam' ingestion somehow or send malicious log/evidence data that somehow screws KEEN up?

I use Qubes OS, so I already have a convenient way to give a test client and a staging server their own machines. This also makes it cheaper than having to rent yet another VPS just for the security testbed!

The result is a small local security lab with a fairly satisfying command:

tools/staging/run-security-tests

That restores KEEN to a known state, runs a battery of fuzzing tests, saves the service logs, then removes the installation's containers and data. The next run starts from the same golden dataset.

Fun!

Two standalone qubes

I use two standalone VMs:

Standalone VMs keep their own root filesystem changes. That suits a lab with Docker, certificates, browser dependencies and local working trees. The qubes persist between runs; the staging script rebuilds the application environment inside them.

The staging checkout lives at /opt/keen. The testing qube has a checkout too, so it can run the matching version of the security suite. Keeping those revisions aligned matters when the API gains routes or changes behaviour.

The ordinary KEEN containers run inside keen-staging: PostgreSQL, Valkey, the API, UI, scheduler, internal worker, HTTP ingestion worker and the optional IMAP worker. I build KEEN's images from the local checkout. Docker still needs whatever base images and dependencies the build requires.

There are two connections from the testing qube: HTTPS for exercising KEEN, and SSH for restoring it and collecting logs. Qubes' qubes.ConnectTCP service provides both.

The small bit in dom0

In dom0, I give the testing qube permission to request those two services from the staging qube. These lines go in /etc/qubes/policy.d/30-keen-staging.policy:

qubes.ConnectTCP +22  keen-testing keen-staging allow
qubes.ConnectTCP +443 keen-testing keen-staging allow

The arguments restrict these rules to the two destination ports from the first qube to the second.

Qubes documents this mechanism for exposing an individual TCP service between qubes. Here, dom0 supplies the policy decision, while the application traffic travels through the qrexec connection.

The application, Docker builds, browser and test payloads all stay in the standalone qubes. The dom0 configuration is just those narrowly scoped permissions.

This is a useful separation, with an intentional management privilege: the testing qube has an SSH account capable of controlling Docker in the staging qube. Compromising that account gives substantial control over the lab server. Both qubes belong to this testing environment and contain synthetic application data.

SSH setup

In keen-testing, the SSH configuration has a dedicated alias:

Host keen-staging-control
    HostName keen-staging
    User user
    ProxyCommand /usr/bin/qrexec-client-vm keen-staging qubes.ConnectTCP+22
    StrictHostKeyChecking yes

The staging qube needs a running SSH server and an authorised key for that user. The alias gives the runner an unambiguous control connection, independently of the website's hostname.

SSH host-key verification still applies. For the initial connection, compare the offered fingerprint with this command run directly in keen-staging:

sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

Then, in keen-testing, make the first connection interactively:

ssh -o StrictHostKeyChecking=ask keen-staging-control

Accept it after verifying the fingerprint. Subsequent automated runs use the recorded key and batch mode:

ssh -o BatchMode=yes keen-staging-control hostname

I ran into both of the obvious setup mistakes here: SSH wasn't running initially, and then strict checking quite correctly refused a host key it had never seen. Getting this command working first makes the runner much easier to diagnose.

Real TLS on a local machine

I wanted the browser and HTTP client to validate the staging certificate normally. The name I use in this example is keen-staging.mig5.net, but you would use a domain you control if you're setting this suite up yourself.

Let's Encrypt's DNS-01 challenge makes this possible with a private web service. Domain validation happens through a public DNS TXT record.

My DNS is hosted by Linode. Certbot's Linode DNS plugin creates and removes the challenge record through its API. Once Certbot and the plugin are installed in the same environment, the credentials file contains:

# /etc/letsencrypt/credentials/linode.ini
dns_linode_key = MY_LINODE_API_TOKEN

The token needs the DNS access required for the challenge; its scope deserves care because it grants authority over DNS. In Linode's case, only 'Domains' is the needed read/write scope.

The issuance command I used was:

sudo certbot certonly \
  --dns-linode \
  --dns-linode-credentials /etc/letsencrypt/credentials/linode.ini \
  -d keen-staging.mig5.net

I use the resulting certificate on nginx in keen-staging, which forwards requests to KEEN's UI on 127.0.0.1:8081. The repository includes tools/staging/nginx-site.example as a starting point. Adapt its server name and certificate paths, and allow the loopback connection made by ConnectTCP. It replaces the incoming forwarded-address header and preserves WebSocket upgrades. Renewals need a scheduled Certbot run and an nginx reload when the certificate changes.

In keen-testing, /etc/hosts contains:

127.0.0.1 keen-staging.mig5.net

That entry chooses the local tunnel endpoint. The URL still carries the real hostname, so TLS checks the certificate for keen-staging.mig5.net.

The certificate name is public through Certificate Transparency. The local address mapping is a routing choice, with no expectation of keeping the hostname secret.

Tunneling the HTTPS port

An /etc/hosts entry only resolves a name. Something still has to listen on the resulting address!

I initially had the hostname resolving correctly and nothing listening on port 443. Every readiness check failed with connection refused. A socat listener supplied the missing connection to qubes.ConnectTCP+443.

That listener is now managed by run-security-tests. Setting:

KEEN_STAGING_QUBES_TARGET=keen-staging

makes the runner start a temporary systemd service in keen-testing. It binds to 127.0.0.1:443 and forwards connections through qrexec to the staging qube.

The runner checks that the hostname resolves only to 127.0.0.1 and that the port is free. It uses sudo to create the service, runs socat as the invoking user with the capability needed to bind the privileged port, limits child connections, and removes its service during cleanup. An existing manual listener needs stopping before using this managed one.

TLS remains between the test client and the staging TLS server throughout that forwarding path.

Giving every run the same starting point

The host-side script is tools/staging/keen-staging. Its destructive operations require this explicit marker in the staging checkout's .env:

STAGING_DESTROYABLE=yes

That is an operator interlock. The dedicated checkout, Compose project and VM provide the surrounding separation.

The initial setup, inside keen-staging, looks like this once Docker and the checkout are ready:

cd /opt/keen
tools/staging/keen-staging generate-env https://keen-staging.mig5.net
sudo install -d -m 700 -o "$USER" /var/lib/keen-staging

Review the generated .env, then create the golden state:

KEEN_STAGING_BUILD=1 COMPOSE_PROFILES=imap \
  tools/staging/keen-staging init-golden

generate-env creates fresh lab credentials, including the suite administrator. init-golden starts an empty installation, runs migrations, seeds the demo dataset and adds stored evidence files. It then snapshots the database and artifacts under /var/lib/keen-staging/golden.

There is enough data to give the tests something meaningful to work on: frameworks, controls, assets, risks, documents, objectives, audits and evidence. Text, JSON, PDF, HTML and CSV artifacts exercise download and preview paths too. As the demo seeder grows, rebuilding the golden state brings those examples into subsequent runs.

Snapshotting stops the running application writers while it saves the PostgreSQL dump and artifact archive. A manifest records the schema revision, and checksums are verified before restoration.

Each ordinary restore removes the previous Compose containers, network and named volumes. It restores the golden database and evidence, starts the application, and lets pending migrations run. Valkey state and the scheduler's persistent volume start fresh too.

The VM operating systems, source checkouts, certificates, .env and golden snapshot persist. This reset covers the application data managed by these scripts. A suspected escape into the staging host would require rebuilding the qube as well.

Wiring up the runner

In the testing checkout, install the suite's environment:

python3 -m venv tools/security_e2e/.venv
tools/security_e2e/.venv/bin/pip install \
  -r tools/security_e2e/requirements.txt
tools/security_e2e/.venv/bin/playwright install chromium

Playwright may also need OS libraries for Chromium. The runner looks for that virtualenv's Python directly, so shell activation is optional.

The configuration goes in ~/.config/keen-staging/config:

KEEN_STAGING_URL=https://keen-staging.mig5.net
KEEN_STAGING_SSH=keen-staging-control
KEEN_STAGING_REMOTE_DIR=/opt/keen
KEEN_STAGING_QUBES_TARGET=keen-staging

This file is sourced as shell configuration, so it should be writable only by its owner. The testing qube also needs socat, qrexec-client-vm, SSH, curl, Python and the systemd/sudo tools used to manage the listener.

My runner's SSH invocation sets KEEN_STAGING_BUILD=1 COMPOSE_PROFILES=imap on the remote command. That makes it build the local KEEN checkout and include the IMAP worker each time. It was an important adjustment after the first attempt tried to pull my unpublished application images from GHCR.

Now the single command does the whole cycle:

1. Start the local HTTPS tunnel.
2. Restore the staging host over SSH.
3. Wait for the HTTPS health endpoint.
4. Read the lab administrator password over SSH and run the suite.
5. Save junit.xml and the container logs under staging-results/<timestamp>/.
6. Tear down the staging installation and stop the tunnel.

The runner passes the target confirmation and synthetic-write flags required by the suite. Its exit handler also attempts cleanup after an interruption or failed restore. Errors during cleanup still need attention; the script reports them.

A fuzzer that knows about KEEN

The useful part is the application's security model being expressed as executable expectations.

The suite reads KEEN's OpenAPI schema to discover operations (another fun fact: FastAPI makes it super easy to generate OpenAPI specs - no crazy annotations needed, which is not the experience I've seen with PHP!).

An access-policy inventory assigns rules to those operations, and a backend test requires newly added routes to have an entry (this helps prevent me forgetting to write a test for a new route!). The deployed test then changes a probe user's permissions and checks permitted and refused requests against that inventory.

There are browser tests too. They store synthetic hostile markup in records, open the corresponding pages in Chromium, and inspect the DOM for injected elements, unsafe links and script execution. Checking for injected elements is useful even when CSP blocks a payload's script.

Other modules exercise session renewal and logout, CSRF and Origin checks, redirects, uploads, download headers, malformed JSON and concurrent use of one-time credentials.

This complements the CI tests by checking how the deployed components behave together. It also makes for useful regression testing after a manual security review: turn the finding into a reproducible case and keep running it.

I know and have used tools like ZAP, of course. The difference with this bespoke test suite, though, is that it is very 'KEEN-aware': it knows the layout of the app and what to target, what workflows to follow. I could spend a long time customising that in ZAP too, but honestly, this was faster.

How random is it?

I didn't want the test to just throw the same inputs every time - that would eventually lead to a false sense of security.

The parameter fuzzer uses a fixed corpus of awkward values: oversized numbers, unusual Unicode, malformed dates, traversal strings, markup and other inputs that tend to expose assumptions. It samples and shuffles combinations using a configurable seed.

The default seed is 1. With the same schema and suite, that gives a reproducible selection. Per-run record identifiers and markup markers change separately.

I added this to my .bashrc to choose a seed when I open a shell:

export KEEN_E2E_SEED="$(python3 -c 'import secrets; print(secrets.randbits(32))')"

That gives each new interactive Bash shell a seed. Repeated runs in the same shell reuse it. I can check the current value with printf '%s\n' "$KEEN_E2E_SEED"; saving that value with the results makes a failure easier to reproduce.

For an explicit seed and a larger query-fuzz budget:

KEEN_E2E_SEED=7 KEEN_E2E_FUZZ_CASES=3000 \
  tools/staging/run-security-tests

The case budget caps the query-fuzz plan, whose size also depends on the available parameters. Other modules have their own loops and bounds.

For a run with a seed recorded alongside its reports, I can use:

results="$PWD/staging-results/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$results"
printf '%s\n' "$KEEN_E2E_SEED" > "$results/seed.txt"
tools/staging/run-security-tests --results "$results"

That assumes the runner config leaves the exported seed intact. I also keep the source revision with a finding, because the same seed against a changed route inventory can select different cases.

Keeping a failure around

For cases where a failure occurs, there is this flag:

tools/staging/run-security-tests --keep-on-failure

This allows you to rerun a specific module against the retained state:

tools/staging/run-security-tests --no-restore --keep -- \
  test_09_access_policy.py -x

The managed tunnel is stopped when each runner invocation finishes, including a retained run. The next invocation starts it again. To browse the retained installation between runs, reopen a tunnel separately.

When finished, tear down from the staging checkout:

tools/staging/keen-staging teardown

A lab worth keeping

I like how little ceremony there is once this is wired up. Change KEEN, run one command, get a fresh installation with meaningful data, and exercise it from another machine. A failure comes with logs, a route or test case, and the option to preserve the state for investigation.

There is still, of course, plenty for manual review and independent penetration testing to find. There Will Be Bugs. Every new feature gives me more behaviour to describe, more boundaries to check, and more interesting things to throw at my own application. It's probably never going to be perfect, but it's getting better all the time!

KEEN is open source software - you can get it here . You can try a 30-day free trial of it (with demo data seeded into it) at https://keen.cloud .