Thoughts on OpenBSD manageability

I care about man­age­abil­ity because it directly influ­ences my job and defines how much time I have to spend on triv­ial tasks like patch­ing or upgrad­ing oper­at­ing sys­tems. My goal is to spend as lit­tle time as pos­si­ble doing triv­ial & repet­i­tive stuff so that I have more time for things that I con­sider more impor­tant, like improv­ing our net­work, test­ing new solu­tions or drink­ing cof­fee :)

I use OpenBSD exten­sively for many appli­ca­tions, most of them being net­work related stuff and have done so since the 2.9 release. OpenBSD is ultra secure, rock solid and the ulti­mate Swiss Army knife of net­work­ing in the UNIX world if you ask me. It has *very* few flaws but there are sit­u­a­tions where it is not fea­si­ble for me to use OpenBSD.

We have a lot of servers at work and a good per­cent­age of those are run­ning some kind of UNIX deriv­a­tive. When deal­ing with a large machine park it becomes appar­ent that one needs to find an effi­cient & reli­able way to man­age them. It is pre­cisely here OpenBSD fails for me.

Sev­eral peo­ple have asked me why I am not push­ing for more OpenBSD instal­la­tions at work. The answer is that I sim­ply can­not afford to prac­tice oper­at­ing sys­tem evan­ge­lism at work — it’s about find­ing the right tools for the job at hand no mat­ter my per­sonal pref­er­ences might be. Period.

In gen­eral, the meth­ods described below work well and are very reli­able but they do not scale well thus lack­ing the desired effi­ciency when deal­ing with many machines.

My main con­cerns are the following :

  1. Short life­time of releases.
  2. Upgrad­ing & patch­ing is cum­ber­some and incoherent.

1. Release lifetime

OpenBSD sup­ports releases for 1 year and once a release reaches its EOL no more patches are pro­vided. That means that one must update all servers run­ning OpenBSD within 1 year from the instal­la­tion date to keep OpenBSD sta­ble and secure. Since skip­ping releases dur­ing upgrades is not rec­om­mended you must either upgrade your boxes every 6 month (when a new release come out) or alter­na­tively skip a release an then rein­stall the oper­at­ing sys­tem with the newest release to bring your boxes up to date.

Even though installing OpenBSD is fast and easy — imag­ine hav­ing to do this on 150 or 200 servers at least once a year.

2. Upgrad­ing

Upgrad­ing from release to release is some­what cum­ber­some and time con­sum­ing. You can either rein­stall the entire oper­at­ing sys­tem or install / upgrade the OpenBSD source code and then com­pile the new release. Once you have upgraded the oper­at­ing sys­tem you still have to merge changes between the old and new con­fig­u­ra­tion files and update any appli­ca­tions you might have installed through ports(7) or packages(8).

Again, not very effi­cient for upgrad­ing 150 or 200 servers at least once a year.

2.1 Patch­ing

There are a num­ber of ways to patch OpenBSD :

Using errata paches

You can keep track of issued patches via the OpenBSD errata page and down­load the indi­vid­ual source code patches from there. Every patch comes with a detailed descrip­tion of how to apply them. Once you obtain a patch you must apply the patch to the OpenBSD source code and then rebuild and rein­stall the affected file(s). Errata patches are for the base oper­at­ing sys­tem only.

Fol­low­ing –STABLE

You can upgrade & fol­low STABLE, which involves down­load­ing or updat­ing the entire OpenBSD source code and com­pil­ing the whole oper­at­ing sys­tem on a reg­u­lar basis. While being a triv­ial albeit time con­sum­ing process, it involves a reboot of your server and takes up con­sid­er­able resources dur­ing the com­pile which is not always what you want. On the other hand, fol­low­ing STABLE is easy to auto­mate and can be done using lit­tle human intervention.

Using release(8)

By using release(8) you can update one machine to STABLE and use that machine to build a com­plete binary ver­sion of OpenBSD that you can install on other OpenBSD machines of the same hard­ware archi­tec­ture. The ben­e­fit here is that you free up the resources needed to down­load & com­pile OpenBSD on every machine. The big prob­lem with release(8) is that it equals a full install which is dif­fi­cult to auto­mate and thus requires a lot of man­ual intervention.

Bin­patch

Bin­patch is a frame­work to cre­ate bina­ries from patched source code. It can auto­mat­i­cally down­load the source patches pub­lished on http://www.openbsd.org/errata.html, apply them, build them, and pack­age the result into binary patches. These binary patches in turn can be dis­trib­uted across a net­work and applied to any num­ber of servers. I use bin­patch for sev­eral pri­vate machines and it works great but it can be dif­fi­cult to keep track of which machines have which patched bina­ries installed.

There is only one big prob­lem with bin­patch — it is not offi­cially sup­ported by the OpenBSD devel­op­ers which basi­cally means that you are on you’re own if things go wrong. That is rea­son enough for me not to use bin­patch at work.


So far I have described meth­ods for updat­ing / upgrad­ing the base oper­at­ing sys­tem, but many times you will have installed extra soft­ware using either ports(7) or packages(8) and they too must be updated. Ports are updated / upgraded by bring­ing the ports source tree up to date with your cur­rent release. You can down­load or update the ports sec­tion of the source code alone or update the ports sources together with the rest of OpenBSD if you are fol­low­ing STABLE. A spe­cial script can then find and list any out­dated ports — but you must pro­ceed with upgrad­ing each port manually.

Pack­ages are a bit eas­ier, they can be updated / upgraded in binary form using the pkg_add(1) tool.

If you can keep away from ports(7) the update / upgrade process is much more streamlined.

Conclusion

All in all, I find the exist­ing meth­ods for updat­ing or upgrad­ing a large num­ber of OpenBSD machines inef­fi­cient and labor inten­sive com­pared to other solu­tions avail­able. Their main­te­nance requires too much human inter­ven­tion too often.

Much of my cri­tique is applic­a­ble to many other BSD or Linux dis­tri­b­u­tions since they share the same flaws when it comes to man­ag­ing large installations.

For the rea­sons explained above I have begun to deploy Ubuntu’s server release. I found their LTS release matches my needs very well — the release life­time is a stag­ger­ing 5(!) years. Updat­ing involves 1(!) tool, apt-get, to update both the core oper­at­ing sys­tem and any installed appli­ca­tions. Fur­ther­more apt-get can eas­ily be auto­mated. So far, Ubuntu has been nice to work with — it has show both sta­bil­ity, man­age­abil­ity and good performance.

One Comment

  • What hap­pens if you make a mis­take of ker­nel recom­pi­la­tion?
    Or even install any soft­ware from sources? No more auto updates for you. I made this mis­take once with a ker­nel :-(
    Very time con­sum­ing.
    Then I’ve decided to go only with “offi­cial” apps or pack­ets as they are called in Debian. And I can say that apt rulez.

Leave a Reply

Your email is never shared.Required fields are marked *