Source based load-balancing in HAproxy based on X-Forwarded-For header

We had some application servers behind an active/passive HAproxy loadbalancer pair (using keepalived to arbitrate the IP on failover).

We needed to put a WAF product in front of the HAproxy pair (e.g Sucuri's CloudProxy or CloudFlare). This might seem odd to put a reverse proxy in front of a HAproxy pair (yo dawg, I heard you like proxies), but we need to do some funky extra munging of URLs and the like via HAproxy configuration rules, which upstream providers can't account for.

Yubikey 2FA on Qubes redux - adding a backup key

Previously I wrote about adding Yubikey 2FA authentication in Qubes (not for using Yubikey on remote sites, but on 2FA of your Qubes system itself), explaining a couple of the differences in my technique compared to the official docs (e.g I don't believe in backdooring with a password in absence of your Yubikey, especially since with a usbVM, that VM can read the password as you type it!

Monitoring Certificate Transparency logs for fraudulent SSL certs with Scumblr

I read with interest this article by Facebook, about detecting (possibly) fraudulent SSL certificates being issued by CAs.

I wasn't previously aware of Google's Certificate Transparency initiative, but it seems like a good idea. Basically there exists a sort of blockchain of public, append-only logs of all SSL certs that are being issued (at least, where the CA is cooperating to publish that info?).

Verifying your key/identity on with Qubes and Split GPG

You are using Qubes with Split-GPG, but you want to verify your GPG key at, via the command-line with bash and curl (you're not storing the key on Keybase's servers).

The verification command is a curl request but includes calls to the gpg command-line client. You have two problems:

1) Your GPG VM is not connected to the network (for good reason)
2) Another VM that is connected to the network, can't use the gpg command.

Batch updating TemplateVMs in Qubes 3.0

If you are a Qubes user like me, you probably have a number of TemplateVMs which your App/USB/ProxyVMs are based off of (e.g, you aren't using the same template for all VMs, as you would otherwise develop a fair bit of irrelevant bloat across them all).

If so, you've probably discovered that keeping all those templates up to date with security updates is rather cumbersome when doing so manually.

PHP backdoor shells in Drupal: not always file-based

The other day I was trawling through the overnight OSSEC notifications received at a customer's infrastructure and I came across one such item:

OSSEC HIDS Notification.
2015 Jun 21 17:05:58

Received From: (>/var/log/auth.log
Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."
Portion of the log(s):

Jun 21 17:05:57 XXXXXX sudo: pam_unix(sudo:auth): conversation failed


"That's odd", I thought. This server doesn't really get frequent logins let alone use of sudo.

Yubikey in Challenge Response mode with Qubes

EDIT: this is still worth/necessary reading, but see an April 2016 update where I describe further improvements, including supporting a backup key in case you've lost your first.

I bought a new Yubikey for use with Qubes, but I had some issues with the challenge-response instructions in the documentation.

That is:

'So, what is it you exactly do?' - Part six, high availability

In this segment of 'what do you do, sysadmin?' I'll cover the area of high availability, building infrastructure that can withstand failure, and preparing for worst case disasters.

Using Ansible and Jenkins to check for stale inodes

As part of teaching myself Ansible this week, I've been porting some of my sysadmin toolset into playbooks. I thought I'd share one today that I call 'Stale service check'.

Anyone in operations who does patching on a routine basis knows that a simple 'apt-get upgrade' is rarely enough to apply a security update; Linux uses linked libraries, and frequently when a library is updated, many services that depend on that library are not yet using the new version. OpenSSL is a classic example (remember why you had to 'reboot' to fully clear the Heartbleed vulnerability?)