You can’t just buy a car and expect it to work great. Any vehicle needs maintenance across its lifetime. If you run it long enough without changing the oil, rotating the tires, and flushing the radiator, it’ll eventually fail you.
Splunk is like that, too.
Splunk requires maintenance to keep it in good running order. Like buying a car, purchasing a Splunk license is only the first step in an active process that keeps the instrument working well. Engaged maintenance over time will not only keep it running the way you want, but also decrease the cost of ownership across its lifetime. That’s true of both your car and your Splunk.
Not sure if you’re keeping up with Splunk the way you should? Consider two central aspects of maintenance that repay the investment.
Documentation
Good documentation—when technologists write down why they made design decisions—is a mark of excellence in an organization. For understandable reasons, it doesn’t always get accomplished. Tech folks are busy getting things done, after all, and time spent documenting could be applied to the rush of incoming projects. But good documentation is foundational to well-performing IT in general and Splunk in particular.
Does your organization possess standard naming conventions, an essential aspect of good documentation? Splunkers who use ad hoc naming risk confusing those who come after. To keep things clean, publish labeling rules for configuration artifacts, technical items, and different kinds of alerts. Best practices dictate that naming conventions reside in a central location Splunkers know and have access to, often called “the wiki.”
Good naming practice typically moves from general to more-specific designations for easier searching. For example,
<datacenter>_<application>_<hostname>_<process>
begins helpfully with the data center designation, then fills in details in order about the application, then hosting, then process. Holding to this policy across the Splunk environment will reduce confusion and save time.
Training
Splunk systems aren’t the only thing that require maintenance. The people do, too. If you perform your IT work in house, investments in their professional development will pay dividends.
That investment starts at the beginning of your relationship: as you bring new Splunkers or tech personnel into your organization, pay close attention to their onboarding. Actively orient them to your Splunk to maximize the value you get from your new Splunk administrator. A thorough walk-through can take the better part of a day, depending on the system’s complexity. Even after that orientation, give them time to learn the intricacies. It can take weeks for a Splunker to get the hang of a new system, so a “sink or swim” approach will likely frustrate both you and your new hire.
Splunk certifications are an essential part of keeping your organization’s skills up to date. Consider hiring experts who are already Splunk certified, rather than training existing personnel. The reason? Adding Splunk certification to a resume increases an employee's marketability. It’s not uncommon for organizations to train existing personnel, only to have those newly minted Splunkers move on to greener (or at least ostensibly better-paying) pastures.
Besides certifications, make sure your IT experts have time to keep current with new features, too. Every couple of months, Splunk ships new minor versions that may fix bugs or impact existing systems and workflows. Notices come out through email blasts, website notifications, conference talks, and press releases. Keeping Splunkers too busy to check these can slow their adoption and impede your operations. Think about continuing education for your Splunkers, too. Training with operating systems such as Linux or Windows is a good start.
Not sure your documentation or training programs are up to snuff? Think it might be time for a tune-up to keep your Splunk in good working order?
We at Conducive meet you where you are—from a survey of your system to training your personnel to maintaining your Splunk. Give us a call and let’s talk it through.
