Simplify The Management Of Your Frozen Splunk Data With Conducive's Splunk Archiver
Restore, Report and Archive – Is it right for you?
130 and Counting
130 and Counting
Why should YOU use Conducive's Archiver for Splunk?
AFTER A YEAR, YOU WILL DISCOVER THAT THE DISK SPACE REQUIRED FOR FROZEN DATA INCREASES EXPONENTIALLY
Why does required disk space increase exponentially?
- Splunk’s default archiving solution doesn’t remove redundant data from buckets or duplicate buckets from a cluster. Splunk freezes all of your data, even thought that isn’t necessary.
- With the following calculation you can see how easily a 1TB license in a clustered environment (replication factor of 3) can require at least 1 Petabyte of storage every year.
1TB Per Day X 3 (replication factor) X 365 = 1,095 TB per year = 1.095 Petabytes.
Does the Archiver fix this?
Conducive’s Archiver for Splunk removes redundant data and eliminates duplicate buckets.
Our customers are seeing an average of 80% reduction in storage space per bucket by removing redundant data and an additional 66% reduction by removing duplicate buckets.
For a 1TB license, your annual long-term storage requirement should require less than 100TB per year – a 90% reduction from 1.095 Petabytes.
Can you do this without Conducive's Archiver?
You could modify Splunk’s default script to remove redundant data, but that won’t eliminate duplicate buckets. And, when you remove the redundant data you will not be able to track the contents of each bucket when you need to restore data.
YOU WILL REALIZE THAT IT'S NEARLY IMPOSSIBLE TO RESTORE FROZEN DATA
Why is it difficult to restore data?
- Splunk’s built-in archiving solution copies frozen buckets to a directory of your choice and leaves everything else up to you. Splunk doesn’t track frozen buckets or help you restore them. You have to manage the storage space, and you have to find the buckets when you want to restore them.
- A 1TB per day clustered environment will create a minimum of 150 to 300 buckets per day. For 1TB, Splunk will create between 36,000 and 120,000 buckets per year, and in many cases more than that. When you need to restore data, you have to search through each of these buckets (represented as a folder or directory on your file system) to locate the data you want to restore. You also have to ensure you only restore one copy of each bucket. Restoring duplicate buckets could create duplicate search results. Done manually, this could take weeks. Of course, its possible to write a script to identify some of the data, but that requires time and skills.
How does Conducive's Archiver make restoring data easy?
To easily restore frozen data, Conducive’s Archiver tracks all of the details of each bucket, allowing you to restore by host, sourcetype, index and data range – all with the click of a button.
No more searching through thousands or millions of directories to find the data you want to restore. No more ensuring duplicate buckets aren’t restored. No more hassle. Restore your frozen data at the click of a button.
Case
– National Retailer located in the Midwest
Restore Splunk Frozen Data
“We started off using Splunk’s built-in mechanism to freeze/archive our compliance data. What we didn’t realize at the time was how difficult it would be to restore that data.
Our auditors requested that we go through an exercise to prove we could restore data for a specific time period across specific hosts. That’s when we discovered we had millions of frozen archive files in the S3 archive. Because the entire archive was multiple terabytes of data, we we knew we didn’t have enough disk space to restore all of it, which would have been the easy solution. Our goal was to restore the subset of frozen files requested by the auditors, but we calculated it would take at least 6 person-days to identify the files we needed to restore.
We started searching the web for a solution when we found Conducive and their Archiver for Splunk. Using Conducive’s Archiver we were able to scan and catalog our existing archive, allowing us to restore the exact data requested by the auditors, all in less than 1 day.
We’re now using the Archiver to both manage frozen data archiving, as well as using it to provide reports to the auditors and restore the data as requested. We can do all of this from a UI that lets us choose the date ranges, sourcetypes, indexes and hosts to restore. The entire process usually only takes a few minutes of time.”
The Old Way
Search through millions of files to find the frozen data you need to restore.
The New Way
Use Conducive's Archiver to select the date range, source types, hosts and/or indexes to restore.
Archiving Made Simple - IRS Data Retention Requirements
“We started with the idea of using Splunk’s built-in solution, but wanted a more comprehensive enterprise solution that includes compression, encryption and native cloud storage integration. After talking with a few Splunkers, we were introduced to Conducive and their Archiver for Splunk. Conducive’s solution enabled us to easily manage our Splunk frozen/archived data.”
After implementing the Archiver, we reduced our storage costs to about $4/TB/month, or a little over $1000 per month in 7 years – and this number is dropping because cloud storage costs are dropping. We’re also able to easily provide reports to auditors and restore data with the click of the mouse.”
– Government Agency
– Midsized Manufacturer based in the Midwest
Auditor Reporting
“Our internal policy requires that we store 6 years of data, and our Auditors have asked that we provide reports proving this data is available and submit to the occasional test to restore the data. We currently keep 18 months of storage accessible. We don’t want to keep 6 years of accessible data on local disk, and we don’t want to use Splunk’s S2 implementation to move searchable data into the cloud. We’d prefer to compress and archive the data to keep the Auditors happy.
We found Conducive’s Archiver for Splunk by searching the Spunk App Store (Splunk Base). Their solution gives us reporting and restoring, along with managed archiving in the cloud. Additionally we can compress and encrypt the data in transit and at rest.
Using Conducive’s Archiver for Splunk, we are able to provide timely reports to our Auditors and restore data as requested.
We’re now using the Archiver to both manage frozen data archiving, as well as using it to provide reports to the auditors and restore the data as requested. We can do all of this from a UI that lets us choose the date ranges, sourcetypes, indexes and hosts to restore. The entire process usually only takes a few minutes of time.”
Features & Benefits
One Click Restore
- Restore frozen data based on time range, host, sourcetype and index.
- Archiver retrieves selected data from storage with the click of a button.
Managed archival process
- Ensures that your frozen data is properly archived.
- Easily create reports for your auditors.
Reporting
- Provide auditors with reports proving the data is archived/frozen.
- Prove to Auditors that your data is restored.
- Have a reporting solution ready when the Auditors ask.
Compress data to save storage space
- Automatically compress data to about 20% of the original size.
- Reduce storage costs.
Encryption
- Once the data leaves Splunk, it remains encrypted throughout the entire process
- Encrypted at rest and in-flight.
Deduplicate data
- Deduplicate buckets from clustered indexers. (A clustered indexing configuration will contain multiple copies of buckets.)
- Store only one copy of these buckets, saving you time and money
Cloud or on-premise storage
- The Archiver integrates with all major cloud storage provides.3
- Or choose to store locally.
- Storage costs as low as $ for TB per month.
Why buy from you? What makes you so special?
Who are you guys, anyway?
Conducive has been in business since 2006. We are a Splunk Accredited PS Partner, a Splunk license re-seller and technical software developers.
What happens if I wait?
How much does it cost?
Are there any long term contracts?
We require a minimum 3 year contract.
How do I present this to my team?
What does it integrate with?
The Archiver integrates with Splunk as well as all of the major cloud storage provides such as AWS and Google.
