Wow, long lapse in posts there. Sorry folks. On with the show:
Here's the problem. The Drupal security team do a fine job, following up on reports, auditing contributed modules with a point release to check for security weaknesses, working on core. It's all good, except for one thing:
Contrib release process.
Security updates are rarely just security updates! Normally they're a whole bunch of stuff the maintainers were working on, a bunch of other minor bug fixes that have been queued for a few months, maybe some UI changes and anything else that was languishing in the issue queue when the security team came knocking. So what happens? Security hole is patched, patch is applied to -dev, release is rolled pronto and *everything* goes out the door at once, warts and all.
So it's not a 'security update'. It's a module update that happens to fix a security problem. What if you don't want the other stuff? (Sometimes the 'other stuff' will actually even break your site - that's a whole other blog post, but changing mark-up and CSS in the same branch as a seemingly innocuous 'update' breaks people's websites. Don't do it!) What if your site is stable and you only want the security update, not the other stuff?
You have to go and dig out the patches *specifically relating to security* and apply them. This is more than just a royal pain. Even when you do that, Update Status will continue to needle you that your modules are dangerously insecure, even though you know that is not the case. Gah!
I am told some enterprise Drupal users (and by users, I mean big teams in the corporate world) disable Update Status completely and manage their own security patching the hard way, simply because they don't trust contrib 'security updates' and won't just blindly apply them. God knows, I've blindly applied them a few times and been bitten - hard!
So what to do?
In the short term, if you're a module maintainer and the security team come along and point out a flaw, don't fix the flaw in -dev and roll a release. That's the worst thing you could do. A hurried release is a mess waiting to happen and an issue queue full of support requests.
Instead, patch the latest point release, fixing the security team's concerns, and release that. Sure, you'll have to copy over your current -dev code to push it out the door, but that's what VCS is for! Once your security release is out, and it *really is* a security release, you can reinstate your 'work-in-progress' -dev code and carry on.
This way you have a release that truly fixes the security problem, and *only* the security problem, and you can properly plan your next release instead of doing it in a big hurry. And if people don't keep up to date with your point releases, then they're on their own when it comes to applying security releases made against newer point releases, but that is fair enough. Just don't *force* people to have to do a 'security update' that contains a load of stuff they might want to roll out more thoughtfully at a later date.
And in the future?
Hopefully the move to Git will make a lot of this easier. Without going in to detail, Git's superior branching abilities should make it far easier for maintainers to cleanly release a security fix without including a bunch of other stuff they're working on (perhaps using rebase). I'm not really clear yet on how the old CVS branches and tags are applied in Git, but perhaps there's scope to add a new tag for developing future changes, instead of using the head, so head and the last point release tag should always match and a 'dev' tag contains feature updates.
These are just thoughts - I'd love to hear yours.
22 comments
Getting the patch that was applied?
Hi,
I agree it can be a patience tester. I'd be quite happy to get the patch applied to the latest module version so that I can fix/apply it to the version I have. Does the security team think that providing that patch in the mail out or linking to it would in itself be a security risk?
@stewsnooze
(copied from
(copied from elsewhere):
During the Drupalcon CPH codesprint, several members of the security team (mebak, pwolanin and killes) worked on the new site where
- reporters can file issues directly against a project
- reporters have access to their filed issues and can track progress
- maintainers have access to their issues to communicate with the team & reporter
[...]
Another change we discussed during the Con, is that we will actively help maintainers to create security releases from code used for the latest supported release. This should improve the quality of contrib security releases, lighten the test load and make upgrades less scary for the endusers.
We will not provide patches.
Contrib branch rules
Contrib branch rules make what you suggest nearly impossible. Trying to undo a bunch of changes in dev to a release and applying a patch and creating a release and then reinstantiating those changes in cvs is very very hard. While what you suggest is good, the only usable way to do it is to allow one more level of branching. Which probably will happen more readily when the git migration is complete.
Not sure
Could you expand on contrib branch rules make what you suggest nearly impossible a little? I haven't needed to try this, as (thus far) I haven't had the security team contact me, but I would try to do this:
- Update code to DRUPAL-6--1-5 (imagining that was my latest release)
- Take a local copy of the code in DRUPAL-6--1-5
- Update code to DRUPAL-6--1
- Take a separate copy of this code
- Copy over the checked out DRUPAL-6--1 code with the local copy of DRUPAL-6--1-5
- Fix the security issue (and create a patch for the issue queue)
- Commit to DRUPAL-6--1 and roll a DRUPAL-6--1-6 release
- Update to DRUPAL-6--1 again
- Reinstate the copy of DRUPAL-6--1 and make the same security fix (apply the patch if it will, perhaps manually)
Obviously this will be a massive pain. And the more complex your modules are, the more of a pain it will be (most of my modules are very simple). But this would probably work, would it not?
Or have I missed some horrible detail/is there a massive flaw in my thinking here?
This will almost certainly be easier with git.
Maintaining multiple branches is a pain in the ass with CVS. Being able to apply, tag and cherry-pick patches becomes second nature once you are up to speed.
Also, you shouldnt be blindly applying module updates ever. This is why you should be using something like drush make to automate your build management.
Process
It isn't so much that there is a security risk in providing just the patch. The problem is that you are now running unreleased software and thus our whole Update Status framework is neutered. The right fix is as Greg suggests. We have to switch to Git, make some process docs, AND educate devs on security release process.
Good points, Greg, and
Good points, Greg, and agreed. I've run into this before, and usually I just evaluate if a security release affects my site or not... if not I'll hold out until there's actually a reason to perform the update.
I might need a little bit of a CVS education to put your advice into practice I fear. : P
So maybe I'm misreading this,
So maybe I'm misreading this, but its not that 'Security releases don't work' but more that 'Module maintainers are doing it wrong'. Correct? Because it seems to me there's nothing in the security release process being addressed in your post. You could have just said 'module maintainers don't understand how to use branches effectively'.
Fair point
Title should've been better. The real issue here is 'Security Issues Don't Exist'! *Because* a lot of contrib module maintainers do it wrong/don't know any better.
I specifically am not having a poke at the work of the Drupal security team and their processes at all (sorry if the title gave the impression I was). Though Stew does make the point below that it would be useful if the sec team included a link to the patch(es) fixing the issue in the email digest to save people hunting for it.
Totally agree with you!
Hopefully git is the game changer here. I don't use security updates directly. Instead I put them some time on my test installs. AND I monitor the issue queue of the module in question to see what broke due to the update. My sites are not that prominent so I can live with some days of insecurity but I guess that is not best practice. ;)
Git
A relevant git workflow:
* Start a feature branch. Work on your new code.
* Have a couple other branches for specific bug fixes, etc.
* Security alert comes along! Fix this on its own branch, too.
* When ready to release, push *only* the security fix to the new release
* Once the rest is stable, push those branches out too.
This also means that it's easy to get the security patch, since it's already an isolated branch.
I know a lot of big-name
I know a lot of big-name Drupal companies who don't upgrade point releases unless they're security releases. So then what do you do? If your security release is 6.x-1.5, you can't roll a diff against 1.4, 1.3, 1.2, 1.1, and 1.0 just because it will make some people's lives easier.
And that's not what I suggested
My suggestion was to roll a security release against 6.x-1.5 (using your example) rather than 6.x-1.x-dev. That was my suggestion. People who don't stay up to date with point releases would (and should, IMO) have to sort themselves out. I am *not* suggesting everyone has to provide security releases for old releases. Just responsible behaviour with the current release.
prepopulate 2.0 for example
The recent "security release" of prepopulate broke nearly every site that had it installed, for this very reason. 2.0 wasn't cooked yet, had massive API and usability changes, but was rushed out as a security fix. Ooops.
More branches - less obscurity
I got bitten by a security update early on in my Drupal life
An issue was fixed in a core security release
http://drupal.org/node/78487
and it broke my site!
http://drupal.org/node/78487#comment-688039
Which just shows how even innocuous looking bugfixes can cause problems.
The official advice on security fixes http://drupal.org/node/101497 says
Which doesn't make it any easier to identify security specific patches - hopefully git will allow people to maintain their fix in a remote branch and merge back into the main repository as part of the release process - allowing more helpful commit messages.
I'd hope that git will generally encourage the maintenance of more versions of projects - allowing for greater stability within sites and less need to upgrade.
And the flip side of this would be using branches to manage feature release versions of Drupal - possibly with a Ubuntu like release schedule...
It does seem that the security team puts some stock in the security through obscurity approach - just look at then difference between the bugtraq and Drupal announcements of the same issue
http://drupal.org/node/731710
http://seclists.org/bugtraq/2010/Mar/43
This would be a good
This would be a good improvement.
Unfortunately, -dev releases never seem to get enough testing, which means that pushing something suddenly to a stable release can be a danger.
It'd be good to have instructions included in the security team's correspondance which guide maintainers to do it this way - after all, this is the way that Drupal is now doing it too.
Poor CVS, so misunderstood
Once again, CVS is getting blamed for something that's not its fault.
At DCSF the Security Team met and hashed out a plan for doing security-only branches for exactly this reason. Changing this procedure does not require moving to Git (not that I'm against moving to Git, in fact, I'm actively working on the migration, I'm just incredibly tired of all the misinformation that's spread about how "you can't do X in CVS").
So, for example, for Drupal core 6.18, we created the DRUPAL-6-18-SECURITY branch directly from the DRUPAL-6-17 release tag. All we committed to this branch were the security patches for 6.18, tagged DRUPAL-6-18 from the end of that branch, and shipped the release. Then, we committed the same security patches to the DRUPAL-6 branch and released 6.19 from the end of the DRUPAL-6 branch along with all the other bug fixes since 6.17. So, sites could decide if they wanted 6.17 + security only (6.18) or 6.17 + security + bug fixes (6.19) and upgrade accordingly. Update status says the right things to everyone involved no matter what course of action they choose.
Contrib can do exactly the same. It's just a question of educating maintainers to do it.
Great example
Thanks for an excellent example of doing this with CVS. That's much easier than what I was going to try to do. If I ever need to roll a security release, I'll do it like that. =)
Release Processes in General
The problem with security releases point to a wider problem with Drupal contrib module releases.
I think a lot of contrib's release issues come down to education. Drupal is an excellent CMS. One of the reasons why it is a good CMS is because it is so easy to get stuff done, which create a very low barrier to entry. I am sure there are a bunch of people who have learnt to hack PHP through the use of Drupal. Not all of these people work in organisations which have a strong history of release engineering and processes.
I think that there needs to be clear guidelines for developers on when something should be marked as beta/rc/stable etc and what can happen at those stages of development. I know of at least one 7.x module which is marked as stable, when 7 core is still in alpha.
In my mind it would be great to see Drupal add an extra degree of precision to version numbers, so a contrib module would become "X.x-y.z.a", X is the core version, y is the module's major version, z is the minor version and a is the security patch level. This way I can be sure upgrading a module from 6.x-1.2.1 to 6.x-1.2.2 the only thing that will change is the security fix - which has been tested. (non security) Bug fixes should go into 6.x-1.3.0 and should go through at least a beta / rc release cycle. All major work especially backward compatibility breaks or significant changes in third party dependencies must be made in the 6.x-2.0 branch.
Yes it is more work, yes it is easier with git, but it benefits users and developers a lot. Such a model would also allow the posting of the patch (or a link to it) for security releases, so people know exactly what has changed, rather than picking through diffs and commit logs.
Over the years I've made mistakes with releases, but I've learnt my lesson. These days I fully understand and support Debian's stable policy. It would be good to see Drupal start to move in a similar direction for the same reason.
I agree with Heine
I agree with Heine, that the security team should help the maintainer to create a branch from the latest recommended release that only contains the security fix.
Update Status suggestions
How about "Update Status" having both a note and a silence flag for updates.
This would allow us to evaluate the "security update" and note that we will wait until a later time to update this module.
We commonly ignore "security updates" because the vulnerability does not apply to our sites configuration. It would be nice to log that that the analysis has been done and silence the NAG emails for those modules. This would allow us to still use the Update Status module for notifications.
Greg
This bugs me as well
Just recently there was a security fix for Lightbox2 module, and it got bundled with *a lot* of changes and released that way. So, instead of just fixing security hole, we had to modify CSS and test in every browser.
I really don't understand why contrib devs couldn't provide just patches for certain secuirty holes, instead I have to dig down and manually create them. Also, I think too that Git will be very helpful in this regard.
Post new comment