It’s not uncommon for data centers to spend hundreds of thousands of dollars a year maintaining vital equipment.  A typical maintenance budget can run 1 percent to 3 percent of your initial capital investment for each year of operation, and this amount goes up as the equipment ages.  It’s a necessary evil if you expect high levels of reliability in your data center operations.  But curiously, in this industry (and many others, by the way) the budget allocated to maintain their most important asset is usually a mere fraction by comparison.

Are maintenance budgets misallocated?

It’s well known that human-caused downtime is one of the largest reasons for downtime – if not the largest reason.  It’s just an observation, but oftentimes in my personal experience, I see an allocation of budgeted resources that doesn’t align with what we say is important.  Many budgets look like this:

• Equipment maintenance budget – $600,000/year
• Training for technicians budget – $10,000/year (if that)

Looking at these numbers, it isn’t hard to understand why the majority of downtime is human-caused.

Our technical personnel are just like equipment when it comes to output and efficiency.  We spend tremendous time and effort to find people with the right skills, abilities, and attitudes.  But then after a short orientation, we expect that they will work forever at a high level.  Would you – could you, even – expect a major, expensive piece of equipment to do the same?  People, like equipment, need “maintenance.”  Maintenance for people takes the form of training and practice.

“What gets inspected, gets respected”

That’s a saying we had in the Navy, and it rings true.  When you inspect how people respond to drills, people practice the drills and they become better prepared.  When you inspect the level of power plant knowledge, they put on classes and the overall knowledge level is increased or, at the very least, maintained.  But if you notice, it’s an action that we as leaders must initiate.

If you want your people to maintain their skill and knowledge levels, you need to do two things:

• Inspect it (evaluate, measure, observe)
• Provide resources to improve it (funds, training, drills, etc.)

On the subject of inspection, how much time as a leader do you use to inspect your operations, evaluate the readiness to respond to an upset condition, or test knowledge of plant systems/operations?  I recommend that if you run mission critical operations, you need to be inspecting constantly.  Inspection doesn’t need to be formal, but it does need to be often.  However, when it comes to formal inspections, I would inspect something at least once a week, whether it is conducting drills, examinations, equipment inspections, or document inspections.

How training should be funded

As far as allocating resources in support of maintaining your personnel, I would recommend that your budget be at least one tenth of what you spend on equipment maintenance if your staff runs 24/7.  So your training budget, using the example above, looks like this:

• Equipment maintenance budget – $600,000/year
• Training for technicians budget – $60,000/year, or about $6,000/person/year
  (10 percent of the equipment maintenance budget)

This may seem like a lot when overall company averages are somewhere in the $1,000/year/person level, but your operations are not “average.”  A mistake made by the “average” employee doesn’t cause grave harm or shut down your operations.  Imagine giving our SEAL teams only the “average” budget for training.  What kind of results would we expect, and what kind of skills and abilities would we expect them to have after a couple years of funding at “average” levels?  Could we depend on them as we do today?

As I state in my book (Building Mission Critical Facilities Organizations), mission critical operations require heightened levels of skill and ability.  Your mission critical personnel are the “pointy end of the spear,” if you will, when it comes to your operations.  To maintain these levels of reliability, you must invest in training and practice.  There is no way around it.  The real question is:  What are you willing to spend to eliminate the largest cause of downtime to mission critical facilities?  Where are you focusing your resources?  Are you maintaining your most important assets, your people?