Large scale disk-to-disk backups using Bacula, Part V

The inte­gra­tion of Bac­ula into the rest of our com­pany has been a very straight­for­ward process.  Amongst other things we have an inter­nal con­trol panel that is used to con­duct most busi­ness oper­a­tions like cre­at­ing and chang­ing cus­tomer sub­scrip­tions  plus a billing com­po­nent which con­tin­u­ously mon­i­tors resource con­sump­tion and informs our cus­tomer data­bases about changes in ser­vice plans and so on.

Now, we wanted to inte­grated job track­ing into our con­trol panel. The details would be exposed on a per client basis, enabling both staff, resellers and part­ners to mon­i­tor and check the backup sta­tus with­out going through our front-line sup­port or both­er­ing our tech heads.  Fur­ther­more we wanted to be able to check resource con­sump­tion (disk usage) on a per client basis to enable auto­mated billing. Cus­tom report­ing and trend­ing were also on our wish­list and so are nicely for­mat­ted email noti­fi­ca­tions for our customers.

Almost every­thing we needed was already in the Bac­ula Cat­a­log data­bases. The only thing we had to do was to add

cat­a­log = all, !skipped, !saved

to our Mes­sage resources in the direc­tor con­fig­u­ra­tion in order to store job logs in the data­bases as well. It took me about one day to research and less than a day to write a sim­ple REST API that exposes this data as XML or JSON, depend­ing on what the client wants. The Bac­ula DB schema is rather sim­ple once you find your way around. Inte­gra­tion into the con­trol panel was done in lit­tle more than a day.

An exam­ple of the gen­er­ated XML can be seen here :

<bac­ula>
 <client>
  <jobid>63970</jobid>
  <job­name>x.x.x.x</jobname>
  <level>F</level>
  <client­name>x.x.x.x</clientname>
  <start­time>2010-12-14 02:22:37</starttime>
  <end­time>2010-12-14 05:39:15</endtime>
  <job­files>344002</jobfiles>
  <job­bytes>116245957721</jobbytes>
  <jober­rors>0</joberrors>
  <job­sta­tus>T</jobstatus>
  <dura­tiontime>03:16:38</durationtime>
  <job­sta­tus­long>Com­pleted suc­cess­fully</jobstatuslong>
 </client>
</bacula>

We will expand this in the future. I already have a pro­to­type that enables us to cre­ate, mod­ify and remove clients and their con­fig­u­ra­tion as well as restart­ing failed job and so on  directly from our con­trol panel . It also uses col­lected met­rics  in order to allow the sys­tem to place new clients on the least utilised direc­tor / stor­age servers automag­i­cally, thus enabling an equal dis­tri­b­u­tion across our Bac­ula infrastructure.

Update

Part VI is now online

No Comments

Leave a Reply

Your email is never shared.Required fields are marked *