The other day I was trawling through the overnight OSSEC notifications received at a customer's infrastructure and I came across one such item:
OSSEC HIDS Notification. 2015 Jun 21 17:05:58 Received From: (XXXXXX.example.com) 18.104.22.168->/var/log/auth.log Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system." Portion of the log(s): Jun 21 17:05:57 XXXXXX sudo: pam_unix(sudo:auth): conversation failed --END OF NOTIFICATION
"That's odd", I thought. This server doesn't really get frequent logins let alone use of sudo.
Here's a scenario...
At 4:30AM every Thursday (sysadmin's time), a server's site suddenly spikes in load, because a full backup takes place at such a time, which is not an off-peak time in terms of traffic due to international visitors.
A bunch of users visiting a site on that server receive a flurry of 502 errors trying to load some content - a form of application timeout due to the taxing effect on the CPU related to the backup process.
This article is third in a series of long, windy answers to the inevitable 'but what exactly do you do as a sysadmin consultant?' question. I started writing this because it's hard to give a sufficient short answer.
After reading the SecureDrop security audit announced today, I noted that they GPG-encrypt their OSSEC mail to add an extra layer of protection over the incidents that OSSEC finds and sends alerts for. Neat idea, it never occurred to me. Even though my servers use TLS to transmit mail around, and that I run my own mail server, that traffic still has to hop through some public routes, so why not add more encryption.
I recently spoke at the Drupal Melbourne July meetup about the open source intrusion detection system OSSEC, and how it can be used to give you more insight into attacks/strange/broken activity on your Drupal sites (as well as more generally across your infrastructure).
I really like OSSEC, the open-source intrusion detection system, and deploy it wherever I'm working. Not only is it great from a security point of view (detecting brute force attacks, crawlers, XSS injection attempts, bad permissions on files, modificatons to files, notification of installed/removed packages, presence of rootkits etc etc), but it's also really good at exposing the general state of things on your infrastructure that might otherwise go unnoticed (even if they're logged).
I haven't got around to packaging OSSEC for Debian yet - mainly because I haven't decided how to handle the fact that OSSEC uses a server->agent model that depends on the generation/importing of unique keys for communication (not unlike Puppet with SSL certificates), from an automation/Puppet perspective.