A few years ago, we started working with a Splunk customer with a 10TB license. The customer was on the verge of eliminating Splunk from their environment because they said Splunk didn’t work. Users weren’t receiving their scheduled reports and ad hoc searches took too long to execute. We were asked to help resolve the problem.
We discovered two main issues. They didn’t have enough indexers to support a 10TB license and they had over 100,000 searches all scheduled to run at midnight. Splunk simply cannot execute 100,000 searches at the same time, which meant that users never received their reports in the morning.
Solving the problem for this customer took several months of painstaking work. We categorized each search, found its owner and determined if the search was still necessary. We discovered that most of the scheduled searches were no longer required and reduced the search load down to a manageable number. For the searches that remained, we were able to spread out the execution times to balance the system load. Today this customer is very happy with Splunk and has expanded to a 30TB license.
Chances are good that your environment doesn’t have 100,000 scheduled searches, but it’s likely that most of your scheduled searches run at midnight because that’s the default time to run a search. As your system grows this can become a real problem.
Unfortunately, managing your scheduled searches is still overwhelmingly a manual process. Download our list of reference searches for more tips on managing your Splunk scheduled searches. Here are a few ideas that can help.