As is good practice, on internal mission critical systems that are accessible from the Web, we run automatic security patching utilities (like yum and Microsoft Update).  For example, we are currently running (and enjoying) Zimbra 4.5.10 OSS as our mail server on a CentOS 4 box.  Every night our “yum” routine installs any security patches that is available, and we try to keep up to date with every stable Zimbra release.

However, in the past couple of months odd things began to occur.  The ability to edit iCal events from Sunbird or Lightning started to fail, and in trying to fix that, an upgrade to Zimbra 5.0.1 completely failed (with LDAP and Perl errors).  As it turns out, our yum automatically upgraded Perl from 5.8.5 to 5.8.8, which cased incompatibilities with some key Perl modules.  Once I rolled back to Perl 5.8.5, the odd behavior ceased.

So how do you balance system security and system stability?  Proper Configuration Management procedures baseline versions of software packages together so that we can test security patches and other upgrades on an integration environment before rolling out to production.  So, the ideal software engineering solution is to create an integration environment a for mission critical system, monitor and test updates on the integration site, and when cleared, baseline and promote the updates to production.  The timing of the baselines and promotions would need to be based upon the urgency of the security patches.

The problem with the ideal solution is that it takes a large amount of monitoring and control to be notified when security patches occur and then to baseline, test, and promote.  For our development projects with multiple environments, this is exactly what we do and budget for the time in our project schedules.  However, for our single environment internal systems, the beauty of yum and Windows Update is that can be completely automated.  So can another compromise be found?

I believe that the answer lies with the question: Against what are we trying to protect?  From the perspective of our internal systems, our primary risk is automated vulnerability testers used by script kiddies.  Primarily, they can only hack that which they have direct access.  So one approach is to restrict yum to only those packages that have externally accessible interfaces (in our case ssh, mail, and httpd).  That way we’re not rudely awakened by another Perl update (unless there is a dependency).  It is true that an internal package could unveil a security vulnerability through a complex interaction with an external interface, but automatically updating only external interfaces seems to strike a balance in trying to simultaneously avoid being hacked and accidentally breaking services.