Aegir bulk backup script updated

Submitted by mig5 on Mon, 20/09/2010 - 20:36

Tags:

I once wrote an Aegir bulk backup script that was pretty rudimentary, (although popular for some reason): it interrogated the Apache vhosts of your Aegir system and generated a 'provision-backup' task against each site that it found.

The result was a tarball of the site files and database dump in your /var/aegir/backups directory.

It worked pretty well for its time, other than the issue that plagued us in those days: tasks like provision-backup, which were solely 'backend only', were unable to be fetched from the frontend.

In otherwords, you could backup your site, but this generated backup could not be used with the 'Restore' task because Aegir was unaware of that generated backup as it hadn't been created through the frontend and thus hadn't been saved to the Aegir database. The perils of 'everything being a node' when you do something that isn't!

Fortunately, thanks to Drush Aliases, this is a thing of the past! Now that our backend is capable of properly referencing a site from its alias, we are able to enqueue certain tasks from CLI as if we had clicked the button from the frontend.

See this comment from Adrian in a ticket:

in head you can now do :

drush @hostmaster hosting-task @site.com verify , and so forth.
if the task doesnt exist, it will create it.

This is much more efficient, because in the case of a backup, since it is a 'hosting task' (frontend) as opposed to a 'provision task' (backend), it correctly saves to the database the fact that someone made a backup, and thus this backup can be used to Restore a site to this point in time through the GUI.

It also simplifies the cogs of the script itself, since there's no reason to clumsily look at apache vhosts (especially helpful now that Aegir is multi-server aware)

You can download the new script here. Simply execute it as 'sh backup.sh.txt' as the aegir user on your system (and thus put it in a crontab for regular automated backups!).

This is designed to only work with the latest 0.4 alphas and the latest version of Drush Make.

Let me know in the comments if you have any troubles with the script (I haven't used it with remote sites, since I have none, but I've no reason to think it won't be any different)

Comments

Attiks

Wed, 22/09/2010 - 19:02

I just gave it a try, but I get "This task is not valid" for all the sites, I checked the script and enabled debugging, but I don't see anything strange.

I'm using alpha13

I outputted the command and it contains @@
 /var/aegir/drush/drush.php @hostmaster hosting-task @@bug414562.ubu.attiks.office backup

so I changed the script and removed the extra @, and now it's running

Brian Wood

Thu, 23/09/2010 - 02:52

Hey Mig,

Works great.

I was just looking at the possibility of generating an email if a backup fails. (I can't rely on the Drupal Team remembering to look at the task queue every morning.) Is it possible to get hosting-task to append '-d' to the provision-backup command? I don't see an obvious way to do that... That would allow me to grep the output of the script for '[error]' and kick off and email to the drupal admins.

A more graceful approach might be to look for the error info in hosting_task_log after the backups run. (I normally use a php script when I need a mysql query.)

Any suggestions?

mig5

Mon, 18/10/2010 - 13:42

Hi Brian,

Good question - I was wondering, if you set DEBUG=1 in the script (which adds -d to hosting-task backup), do you get the errors that way?

Or are you saying the errors are missed because the debug is occurring at 'hosting-task' as opposed to at the actual backup command.

Also note that Antoine recently fixed a regression that provides verbose output on mysqldump failures << not sure if this might help (i.e maybe in the next release you'll get more info from a failed backup in this way)

We can't really add -d to the generation of the 'provision-$task' command that gets sent to the backend because it's abstracted to such a point in Hosting, that it would apply to *all* tasks sent by the frontend.

However, I am considering a setting in /admin/hosting/ somewhere where you can 'enable debug mode' to get much more verbose task logs. Would that be useful?

Brian Wood

Tue, 19/10/2010 - 08:25

Rules and Token?


Or are you saying the errors are missed because the debug is occurring at 'hosting-task' as opposed to at the actual backup command.

That's what I was trying to say.

I was just looking at what it would take to send an email when a backup task fails using Rules and Token. I see that sub tasks are added to a task node through node revisions. When a sub task has task_status > 1 it seems to be an error.

Using Rules:

EVENT:
"content is going to be saved"

CONDITION:
"If created content is Task"
and
"numeric comparison" [hosting_task:status] > 1

DO
"send email to arbitrary address"
"log to watchdog"

This token would have to be created [hosting_task:status]

Using "content is going to be saved" is imperfect since this will result in 1 email for each node revision with a failed status.

Brian Wood

Fri, 24/09/2010 - 12:25

> for site in `$DRUSH sa | egrep -v "(^@platform|^@hostmaster|^@server_|^@self$)"`; do

What was your thought in excluding hostmaster? Could something go amiss if we attempt to backup the hostmaster site?

I notice that hostmaster can't restore itself--at least not via the UI--so these backups would have to be restored manually.

mig5

Mon, 18/10/2010 - 13:44

Hey Brian,

Good call - I don't really know what my thought was there :) There's certainly no harm in backing up the hostmaster frontend site like any other site (albeit the caveat that you're aware of, that you'd need to do a manual restore).

I'll modify the script to include backing up the @hostmaster site as I see no reason (now) not to include it. Thanks for the catch.

Steven Jones

Tue, 14/12/2010 - 05:44

If you want a (slightly) configurable solution, then the following is a simple step toward proper automated backups.

http://drupal.org/project/hosting_backup_queue

Add new comment