Drupal And Linux, A Deployment Script
Quick post today, no politics *whatsoever*. ;-)
I just wrote another little batch script for deploying my Drupal sites from local to stage. Before I post the code, here's the workflow here in our office at CMS Professionals:
- Code is stored in Subversion away on the Internet (svn.cmspros.co.uk)
- Each developer has a working copy on their laptop
- Database during development is held on central (dedicated) CentOS/MySQL machine on our office LAN
- Unstable client preview/test versions of the code/database are kept on our CentOS test servers, again away on the Internet (testN.cmspros.co.uk)
The Linux shell script I've written, which I share below, is for deploying from our office to the test server. These are the steps it goes through:
- Dumps database to disk locally and bzips it (you must specify host - can be localhost, in our office it's usually 192.168.1.5 - the dedicated MySQL box)
- Copies bz2 file to test server
- Deflates same bz2 file on test server, ready for importing
- Drops and re-creates the database on the test server
- Restores the database from the SQL file we copied up and deflated
- Deletes local and remote copies of the SQL file (we're done with it now)
- Does an SVN update of the application to latest HEAD code (relies on SSH key-based authentication to be properly configured for this)
- Switches to the application directory on the remote server and does a drush cache clear
Cool, huh? Here it is - just save it in a .sh file and make it executable:
if [ $1 = "--help" ];
then
echo "
#########
# USAGE #
#########
Syntax for running this script is:
./database_push.sh sshuser sshdomain remotedatabaseuser databasename remotedatabasepwd
sshuser - string, username for ssh connection
sshdomain - string, the actual domain where the remote database is
scptargetdir - string, place on remote server to copy db dump to
databasehost - string, hostname or IP for database server (usually localhost)
databasename - string, name of database
remotedatabaseuser - string, name of remote database user
sshappdirectory - string, location of the web app for svn update
remotedatabasepwd - (optional) string, remote password for the database user
." &
else
echo "Creating db dump for $5 from $4 ..."
if [ $4 = 'localhost' ];
then
mysqldump -u root $5 | bzip2 > dump.sql.bz2
else
mysqldump -u root -h $4 $5 | bzip2 > dump.sql.bz2
fi
echo "Copying database dump to $2:$3 ..."
scp dump.sql.bz2 $1@$2:$3
echo "Deflating remote database dump ..."
ssh $1@$2 "bzip2 -d $3/dump.sql.bz2"
if [ $8 ];
then
echo "Resetting remote db called $5 ..."
ssh $1@$2 "echo \"drop database $5;\" | mysql -u $6 -p$8"
ssh $1@$2 "echo \"create database $5;\" | mysql -u $6 -p$8"
echo "Restoring remote database $5 ..."
ssh $1@$2 "mysql -u $6 -p$8 $5 < $3/dump.sql"
else
echo "Resetting remote db called $5 ..."
ssh $1@$2 "echo \"drop database $5;\" | mysql -u $6"
ssh $1@$2 "echo \"create database $5;\" | mysql -u $6"
echo "Restoring remote database $5 ..."
ssh $1@$2 "mysql -u $6 $5 < $3/dump.sql"
fi
echo "Deleting local db dump ..."
rm dump.sql.bz2
echo "Deleting remote db dump ..."
ssh $1@$2 "rm $3/dump.sql"
echo "Updating remote application ..."
ssh $1@$2 "svn up $7"
echo "Clear Drupal cache on remote application ..."
ssh $1@$2 "cd $7"
ssh $1@$2 "drush cache clear"
echo "Done!"
fiMy TODO for the script looks like this:
- Put a mysql line in there to update the database and replace the Drupal temp directory path (if necessary)
- Add code to rsync the files directory from local to remote
Edit: Another note, for me really. I might mount the MySQL server on everyone's workstation and symbolically link files directories to that machine, so not only is the database central but so is the matching set up Drupal files. Just weighing up if that's going to be more hassle than it's worth. Hmmm...
Another resource
Just found this cool article about using Subversion:
http://developer.r-project.org/SVNtips.html
Most of it we know, but some handy ideas, for setting the svn repository and target directory as Linux variables. Nice!
[gharvey@eeepc ~]$ export REPOS=svn+ssh://my.svn.server/path/to/repositories[gharvey@eeepc ~]$ export WORKSPACE=~/workspace
[gharvey@eeepc ~]$ svn co $REPOS/application_dir $WORKSPACE/checkout_location
Thanks for the script
Thanks very much for the script. This is exactly the kind of automation that will help us out on a day-to-day basis, but that we haven't spared the time to write for ourselves. It's this sort of thing that makes me pleased I'm part of the Drupal community!
As a side note, we have three separate repos: one for core, one for the 'all' modules, and one for sites, using a multi-site arrangement, so only one instance of Drupal core and shared modules exists on disk.
This streamlines the amount of code needed to run each site, makes sure only one version of, say views, is needed (in the 'all' folder), and speeds up the checkout process.
As Greg points out, it makes it incredibly easy to upgrade or switch Drupal. We recently needed to switch from Pressflow back to vanilla Drupal, and it was a single svn switch command which did this for all the sites on the system.
At CA (and now me for my
At CA (and now me for my personal sites) we keep core in a separate svn repo from the client sites directory. That means it's still in version control, and you can do core vs. contrib updates with separate svn pushes. Works pretty well IME.
Brilliant!
This is a *great* idea ... I'm stealing it. ;-)
sites is all I have in my repo - kind of
This is the strategy that I employ for managing my repositories.
each project only really contains:
index.php
update.php
cron.php
robots.txt
.htaccess
and /sites
The rest of Drupal core is brought in via externals to a repository that manages core separately.
Then inside of sites/all all contrib modules are brought in via externals as well. Only custom project-specific modules are in the project's repository.
andre
externals versus vendor branching
Hi,
I started with externals but have now moved to vendor branching which allows me to locally (in the site) edit core and contrib modules that I need to. Yes some need editing. When a new module version comes in I pull it into vendor and merge it into my site. On the merge if I get no conflicts then my edits are basically fine. If there are conflicts then I obviously go hunt. If I just had externals I would have to jump through hoops to edit stuff otherwise
Stew
Sites Separate
Pretty similar setup to those mentioned above I keep:
General Drupal Repository
core -> drupal core gets updated here and then deployed to staging / production - lots of sites feed of this
Site Specific one (each website gets one of these)
sites -> with:
-> patched (contrib modules I changed)
-> contrib (straight from contrib)
-> custom (modules specific to site)
-> views (exported views)
-> cck (exported content types)
db -> keep dbs here although never certain of how useful this is in svn as opposed to any old directory just makes management easier I guess
Really like the idea of a docs one in svn as well. Will have to copy that one :-)
run without mysql root password
Hi Greg,
I have a similar script but wanted to be able to run as a normal user so I cooked up a little delete all tables script which avoids the need to drop and recreate the db.
mysql -BNe "show tables" mydatabase | awk '{print "drop table " $1 ";"}' | mysql mydatabaseNote that I don't even reference the password - instead this is stored in the users .my.cnf file making the script more portable
See also
http://www.practicalweb.co.uk/blog/09/07/23/drupal-live-and-dev-sync
Thanks Sean
For pulling from production to local, see also my earlier blog post:
http://www.drupaler.co.uk/blog/database-download-script/425
Two thoughts
First, I pretty much use the same process, but before I upload the sql dump to the test host DB, I take a snapshot of what was there and throw it into a timestamped file. My feeling is that given how easy it is run run this script, mistakes can happen if the script's pointed to the wrong test host (happened once on a staging server - wasn't pretty...).
Second, not sure what you mean by "politics", but I'm currently debating the "push vs pull" method of test deployments. Personally, I'd rather push from localhost to test, instead of doing a checkout. For starters, I use Git rather than SVN and keep more than just the Drupal root directory in version control. Git pulls are all or nothing, so a pull-based approach would require me sending a lot of "cruft" to the test host. Even when using SVN, I'd just rather not have any test/throw-away host with credentials to my SVN server.
So, I do an SVN export of the Drupal root directory to a timestamped tmp directory locally. I then shoot that timestamped "release" to the test server. Then, after the copy of the DB is uploaded on the test host, I simply change a symbolic link on the test server to point to the new code release.
If something goes wrong as part of the deployment, I can simply change the symbolic link on the test server to revert the code, and/or run the most recent timestamped test DB dump.
Thoughts?
-Sean
ThinkShout.com
Politics
I was just having a jab at myself for being a poop-stirrer in recent blog entries - politics wasn't a deployment-related thing. ;-)
Anyway, to answer your safety concerns, if this is directed at the wrong server the database simply won't exist and nor will the application. A load of errors, but no damage. The dump create and tidy-up bits will still function, and all the MySQL, SVN and drush stuff will fail, noisily, so the person who typed the wrong server name will know it.
I do start from the position that I do not care about the test servers AT ALL. It really doesn't matter a damn if something breaks up there, from a client service perspective. It's just a mild irritation for us.
Regarding version control, I'm really comfy with SVN for now and can't spare the time to try out something else when SVN "ain't broke". At the moment our SVN trunk (or HEAD, I guess, in Drupal CVS talk) looks like this:
trunk
www <- Drupal application
db <- stable database snapshot made with last commit
patches <- any patches we applied with issue numbers in filenames
docs <- deployment notes, special instructions, etc.
scripts <- exported content types, imagecache presets, etc.
releases
1.0.0
* branch of trunk
1.0.1
* branch of trunk
etc.
Some notes:
I'm really tempted, in most cases, to actually only store
sitesin version control going forwards, taking the view that core is wholly disposable. If I want it patched for performance, I'll use Pressflow - if I don't care, standard Drupal. Either way, at the moment we're keeping a bunch of code we don't really need. Just means we'll have to be more careful at update time (right now no code back-ups required, just db ... svn switch to new core, doesn't work? Just switch back! And it *never* hasn't worked.).svn on production
we *only* checkout www on to servers, nothing else, so we can't pull "cruft"
Are you using checkout or export of your www directory for production? If you are using checkout, do you have any concerns with having the .snv directories on your live servers?
Checkout
We're using checkout, and no, the .svn directories don't concern us at all. If you don't use checkout, you can't switch branch, at which point I fail to see the benefit. You might as well just FTP up your files.
Post new comment