I care about manageability because it directly influences my job and defines how much time I have to spend on trivial tasks like patching or upgrading operating systems. My goal is to spend as little time as possible doing trivial & repetitive stuff so that I have more time for things that I consider more important, like improving our network, testing new solutions or drinking coffee
I use OpenBSD extensively for many applications, most of them being network related stuff and have done so since the 2.9 release. OpenBSD is ultra secure, rock solid and the ultimate Swiss Army knife of networking in the UNIX world if you ask me. It has *very* few flaws but there are situations where it is not feasible for me to use OpenBSD.
We have a lot of servers at work and a good percentage of those are running some kind of UNIX derivative. When dealing with a large machine park it becomes apparent that one needs to find an efficient & reliable way to manage them. It is precisely here OpenBSD fails for me.
Several people have asked me why I am not pushing for more OpenBSD installations at work. The answer is that I simply cannot afford to practice operating system evangelism at work — it’s about finding the right tools for the job at hand no matter my personal preferences might be. Period.
In general, the methods described below work well and are very reliable but they do not scale well thus lacking the desired efficiency when dealing with many machines.
My main concerns are the following :
- Short lifetime of releases.
- Upgrading & patching is cumbersome and incoherent.
1. Release lifetime
OpenBSD supports releases for 1 year and once a release reaches its EOL no more patches are provided. That means that one must update all servers running OpenBSD within 1 year from the installation date to keep OpenBSD stable and secure. Since skipping releases during upgrades is not recommended you must either upgrade your boxes every 6 month (when a new release come out) or alternatively skip a release an then reinstall the operating system with the newest release to bring your boxes up to date.
Even though installing OpenBSD is fast and easy — imagine having to do this on 150 or 200 servers at least once a year.
Upgrading from release to release is somewhat cumbersome and time consuming. You can either reinstall the entire operating system or install / upgrade the OpenBSD source code and then compile the new release. Once you have upgraded the operating system you still have to merge changes between the old and new configuration files and update any applications you might have installed through ports(7) or packages(8).
Again, not very efficient for upgrading 150 or 200 servers at least once a year.
There are a number of ways to patch OpenBSD :
Using errata paches
You can keep track of issued patches via the OpenBSD errata page and download the individual source code patches from there. Every patch comes with a detailed description of how to apply them. Once you obtain a patch you must apply the patch to the OpenBSD source code and then rebuild and reinstall the affected file(s). Errata patches are for the base operating system only.
You can upgrade & follow –STABLE, which involves downloading or updating the entire OpenBSD source code and compiling the whole operating system on a regular basis. While being a trivial albeit time consuming process, it involves a reboot of your server and takes up considerable resources during the compile which is not always what you want. On the other hand, following –STABLE is easy to automate and can be done using little human intervention.
By using release(8) you can update one machine to –STABLE and use that machine to build a complete binary version of OpenBSD that you can install on other OpenBSD machines of the same hardware architecture. The benefit here is that you free up the resources needed to download & compile OpenBSD on every machine. The big problem with release(8) is that it equals a full install which is difficult to automate and thus requires a lot of manual intervention.
Binpatch is a framework to create binaries from patched source code. It can automatically download the source patches published on http://www.openbsd.org/errata.html, apply them, build them, and package the result into binary patches. These binary patches in turn can be distributed across a network and applied to any number of servers. I use binpatch for several private machines and it works great but it can be difficult to keep track of which machines have which patched binaries installed.
There is only one big problem with binpatch — it is not officially supported by the OpenBSD developers which basically means that you are on you’re own if things go wrong. That is reason enough for me not to use binpatch at work.
So far I have described methods for updating / upgrading the base operating system, but many times you will have installed extra software using either ports(7) or packages(8) and they too must be updated. Ports are updated / upgraded by bringing the ports source tree up to date with your current release. You can download or update the ports section of the source code alone or update the ports sources together with the rest of OpenBSD if you are following –STABLE. A special script can then find and list any outdated ports — but you must proceed with upgrading each port manually.
Packages are a bit easier, they can be updated / upgraded in binary form using the pkg_add(1) tool.
If you can keep away from ports(7) the update / upgrade process is much more streamlined.
All in all, I find the existing methods for updating or upgrading a large number of OpenBSD machines inefficient and labor intensive compared to other solutions available. Their maintenance requires too much human intervention too often.
Much of my critique is applicable to many other BSD or Linux distributions since they share the same flaws when it comes to managing large installations.
For the reasons explained above I have begun to deploy Ubuntu’s server release. I found their LTS release matches my needs very well — the release lifetime is a staggering 5(!) years. Updating involves 1(!) tool, apt-get, to update both the core operating system and any installed applications. Furthermore apt-get can easily be automated. So far, Ubuntu has been nice to work with — it has show both stability, manageability and good performance.