Zero-touch Drupal deployment with Jenkins, Aegir, Git, Fabric and Drush

Submitted by mig5 on Wed, 19/10/2011 - 18:22

Tags:

This time two years ago, I wrote an article called Drupal deployment and workflows with version control, Drush Make and Aegir. It remains my most popular article on my site, according to my stats.

Since then I have discovered the power of Jenkins - heck, it's almost all I write about now :) - and some other cool tools such as Fabric for programming the execution of arbitrary commands on remote servers with SSH.

Recently for some clients, I've been building Jenkins and Fabric-based infrastructure that auto-detects changes pushed to a git repository from dev, and automatically deploys those changes out to Q/A, stage and live servers. The process seems to have an incredible effect on development teams when they realise they barely even have to SSH to any of those remote servers ever again, and let a robot do all the work. People start committing early, committing often, and I'd almost go as far as to say that it makes rolling out changes *fun*.

Recently I took some time to port my fabfiles to do the same auto-deployment to QA, stage and live using standalone Aegir servers. The paradigm works really nicely because in Aegir, a 'migration' of a site onto a new rolling-release build (platform), invokes a backup + rollback functionality out of the box.

I've done a screencast demonstrating the process of taking a change from (theoretically) a local dev environment to test/QA, stage, and live, all by Jenkins 'discovering' the change to the git repo and rolling out instructions to each environment to generate a new build.

The .ogg video can be downloaded here)

The code I use to do the 'deployments' is opensourced at http://github.com/mig5/aegir_cid - feel free to take it and improve upon it!

If Aegir is not your thing, that doesn't rule out Zero-touch deployments: I work with companies who are implementing this with standalone non-Aegir Drupal environments too. Have a look at my whitepaper on Zero-touch deployment if this interests you.

Comments

Adrian Simmons

Thu, 20/10/2011 - 23:24

Great to see how you've been pushing forward the Drupal dev/stage/deploy process. Another pile of stuff for me to learn when I have time!

Can't believe you're still using that old diagram, but whatever works ;)

mig5

Fri, 21/10/2011 - 07:38

It remains the best diagram I've seen on the subject! :)

eft

Fri, 21/10/2011 - 16:24

Excellent screencast as always, rich in useful tools and techniques

I liked how you used your "classic" deployment post as a starting point and then wrapped things up with a teaser to a future screencast on your mail job for Jenkins. Do you always record voice and video on the fly together?

mig5

Fri, 21/10/2011 - 16:30

Yes, and that's why they tend to be full of 'um's :) no notes, or overdubbing later.

Thanks for the feedback

eft

Fri, 21/10/2011 - 17:35

I didn't notice any awkward pauses or slip ups. It demonstrates that you really know your stuff and are an experienced communicator. Go Green Bee!

Andrew Shooner

Mon, 24/10/2011 - 10:17

Thanks for this great screencast. The fact that you are building on the method described 2 years ago really reinforces its viability. One question that was addressed in the original post comments was that of staging content. Based on this vid, i assume that is something you handle on another level?

I know some updates i stage for clients are primarily content-oriented (nodes). Do you have a tried and true method for that case as well, and if so does it come into play with this process?

mig5

Mon, 24/10/2011 - 10:25

Drupal, like pretty much all CMS or anything with database-driven content, doesn't really have a great solution for staging content - and certainly not through automated means. Nodes are not properly exportable because their nids may conflict with other versions of the sites - this might change in Drupal 8 with the UUID features.

To me this is still a manual affair, or you can augment your approach with node_export or Deployment modules.

Alfred Nutile

Sat, 21/01/2012 - 16:15

Has anyone had luck using say backup_migrate to just migrate non content related tables to the destination site. Leaving then any content table changes like new field etc up to a feature or hook_update_N. So you could push your new settings live with out losing the clients content?

Dane

Tue, 01/11/2011 - 01:48

Very cool! I'd like to implement this system for a somewhat simplified workflow- I don't want to create a separate Jenkins job for every single platform, since I create new platforms all the time for development purposes and my makefiles are all stored in a single repository. Can you give some pointers on how this might be possible? I think Jenkins or the deployment script would have to be able to tell which platforms have changed, and then build only those changed platforms.

Dane

Tue, 01/11/2011 - 04:07

FYI, I describe the modified workflow that I came up with at http://danepowell.com/node/223

I still have to have a separate job for every platform / Git branch, but I don't have to make any changes to the job configuration so it's very easy to clone existing jobs (I just name the project the platform branch name, and this gets passed along to the deployment script). I still need to figure out how to modify this to automatically migrate / install sites as well.

Max

Wed, 30/11/2011 - 09:08

Thanks for sharing the Jenkins auto-deploy solution you have been developing. Do you use a Jenkins script to delete old platforms or do have your clients go in and delete the old platforms manually?

I/we manage nearly 50 sites right now with Aegir with more being moved into the environment and using the system described above, that would require 1 platform per site on dev/stage/prod so there would be 150 platforms. I think that is a bit overwhelming, though with Jenkins doing the cleanup work, it would be manageable.

Thanks for your great work and time to share your experiences.

dominikb1888

Fri, 09/12/2011 - 01:22

For the downwards deployment problem: Would anything speak against a script which combines the information from aegir's drush alias files and the vhost's database credentials to create locally valid aliases on a local dev environment? We're working on the quickstart environment and would create a local development build. Drush quickstart-create allows you to specify a makefile already but doesn't allow you to specifiy a site to sync data from. So adding an attribute to drush qc could solve that problem. Adding something like drush quickstart-sync --domain=, would then allow for syncing current live data for successive developments.

Chris Archer

Sat, 10/12/2011 - 21:41

Just read this article, looks like a really good effect solution to the drupal/aegir staging environment. Look forward to implementing it this weekend. It sounds like this is better suited to larger sites that need their own platform is their anybody using this for platforms that have multiple sites, as we are looking at creating a SaaS for some of out smaller clients. Any suggestions?

Cheers for the great write up.

Chris
SB Design

Johnny Utahh

Wed, 18/01/2012 - 20:28

File metadata/permissions? How does one properly manage consistent/secure file metadata/permissions for auto-deployment? Is this handled by some non-git thing tool/mechanism?

eg: http://superuser.com/questions/367729/how-to-reuse-extend-etckeepers-met...

Add new comment