My last two posts have sparked a lot of interest — my mailbox has been overflowing with lots and lots of questions from all over the world. This post will provide more details about our Bacula infrastructure while the next post will discuss our Bacula configuration.
Let me give the impatient readers a summary : We have applied absolutely no tuning to Bacula itself. A lot of time and effort has been spend on actually trying to understand how it works and how this relates to the resources that Bacula was given in our environment. This has been key to our success with Bacula.
Nevertheless it is kind of amazing that Bacula is able to orchestrate such vast resources in a very efficient manor. Kern and his crew know their stuff, that’s for sure.
I recommend that you go and fetch your favourite beverage before continuing; it’s a rather long post.
Now, we have a total of 9 servers powering our Bacula infrastructure. Three Directors, three Storage servers and three MySQL Catalog servers. As such each Director “owns” a dedicated Storage and Catalog server. Our Directors run Linux and both our Storage and Catalog servers run Solaris.
Let me give you an overview of how our Bacula environment is constituted.
All our machines have access to a special network which is physically separated from our various production networks and 100% dedicated to backup and restore operations. We are talking 100+ switches, dedicated NIC’s, fibers, cables, etc. Each rack has a dedicated switch for doing backup operations. These switches each have a 2 Gbit uplink to a 30 Gbit network backbone (3 core switches with redundant 10 Gbit uplinks each), that also is entirely dedicated to backup and restore operations. The backbone is connected to our remote backup location via a number of 10 Gbit fiber links that in turn provide 10 Gbit network connectivity to each of our Bacula storage servers.
Network operations are highly efficient and are continuously monitored for both throughput and latency. We treat this network exactly like our production networks; solid design, a high degree of redundancy good quality equipment and usable monitoring are essential to its operation. Both network throughput and latency are crucial parts of our Bacula performance.
All our Bacula storage servers run Solaris and all Bacula volumes are stored on three large ZFS storage pools. I have to admit that I am a Solaris fanboy. I could rant for hours about the wonders of ZFS , dtrace, COMSTAR, crossbow and so on but I will reserve this for a future post. For those amongst you that are unfamiliar with ZFS, you really should check it out. ZFS is a modern filesystem invented by Sun that allows you to turn your hardware into extremely efficient and highly scalable storage machines. It offers unprecedented data protection and a wealth of advanced features.
- is always consistent on disk (repeat after me : No more fsck. No more fsck. No more fsck.)
- checksums every block written to disk to prevent silent data corruption
- has self healing capabilities
- utilises a pooled storage model
- has a pipelined I/O engine (massive scalability)
- has no arbitrary limits (supports unlimited filesystems, files, links, directory entries, filesize and so on)
- offers advanced features (caching, inline compression, inline encryption, zero cost snapshots, clones, replication, etc.)
- offers a very simple administration model
I have seen a lot of bad, bad things happen to our ZFS storage servers in the last three years. We encountered defective backplanes, bad cables and memory sticks, faulty controllers and lost a lot of disks all without loosing as much as a single byte of data. If you care about your data you want ZFS.
Now, every storage server has a 120+ disk RAID-Z2 pool comprised of multiple 7 disk wide vdevs, build entirely of 7200 RPM nearline SAS disks. We have three PERC 6/E controllers per Storage server, which allows us to attach up to 9 MD1000 disk arrays. All arrays have dual I/O modules and multipathing enabled.
I spend a considerable amount of time profiling the Bacula storage daemon under production load. Dtrace and Solaris in general were a phenomenal aid in discovering stuff like I/O patterns and I/O sizes, checking for signs of possible lock contention and so on. Solaris provides a lot of transparency compared to other Operating Systems.
We used a lot of this information to run simulations, establishing baselines for testing, etc and discovered a number of things :
- Adding devices to the storage daemon increased parallelism and thus performance.
- No major locking issues were detected when running with 100 active devices (Kern had some concerns causing our investigation).
- The Bacula storage daemon generates asynchronous sequential I/O.
- The recordsize (blocksize) of 64k per ZFS filesystem was most efficient for both backup and restore operations.
- 16 GB DRAM per storage server was a sufficient ZFS ARC size
- CPU utilisation was acceptable.
The Solaris version of the Bacula storage daemon is stable and performs extremely well. We run 100 jobs at a time per Director — Storage — Catalog pair, pumping ~2 GB/s into our storage servers. CPU utilization is < 90% (2 x 3 Ghz Quad XEON’s) and is almost evenly shared by Bacula and ZFS / Solaris.
All three storage servers are scheduled to be replaced with 48 core Dell R815’s. Once we have them in place and have converted our aforementioned backup network to utilise jumbo frames we should get really close to saturating those 10 Gbit links.
MySQL Catalog Servers
Our Catalog servers run MySQL on Solaris for much of the same reasons that our Storage servers do. ZFS provides us with hybrid storage pools capable of enormous IOPS rates, low I/O latcency, easy and proficient snapshotting of our MySQL data and the ability to match the underlying ZFS filesystem recordsize to the acual I/O size used by InnoDB. Dtrace provided us with an insane amount of information about exactly what our databases are doing and how configuration changes influence stuff like query response times and so on.
- The Bacula DB schema and associated queries are not written specifically for MySQL performance
- The Bacula developers have more experience with PostgreSQL when it comes to larger installations.
However, this does not mean that MySQL and Bacula are incompatible. We have some very large jobs (15+ million files and more) which behave fine when doing both backup and restore operations. But hey, perhaps the MySQL force is particularly strong in us ? A search of the bacula-users mailing list will provide the curious reader with more information about our InnoDB and ZFS configuration.
All our MySQL Catalog servers have 2 Quad Core CPUs and 32 GB DRAM. The underlying ZFS storage is comprised of 30 x 750 GB nearline SAS disks in a mirror configuration (mirror beats RAID-Z when it comes to reading data) and a couple of enterprise grade SSDs to facilitate the offloading of synchronous write operations in ZFS. Since almost all MySQL write I/O is synchronous this effectively means that all writes generated by MySQL are terminated against our SSDs instead of our spinning rust.
This concludes the tour of our Bacula infrastructure — next up will we a post about our Bacula configuration.
Part IV is now online