Handling Hundreds of GB of Log Data with Hub

Behind door 6 of our Advent series we have today’s walkthrough on how you can handle massive amounts of log data with Hub:

When you first start out with Gibraltar you usually don’t generate a lot of data - our data files are highly compressed, your sessions in development aren’t too long and there aren’t too many of you.  Then comes the day you launch everything into the wild where people might run your application for days, weeks, months…  Perhaps you have many thousands of users.  Now the data is really scaling up!

Fortunately, the Hub is designed to handle this.  Out of the box the hub operates with a pretty conservative data limit - 2GB.  Once you have more than that amount of log data the oldest data (based on when it was received) is deleted until it’s back within the storage limits you set.  But, say you’re running a private hub and have a few hundred GB of free disk space: let’s put that to work!

This change has to be done directly on the Hub server, so log into the OS on your Hub Server and start up the Gibraltar Hub Administration tool.  Once that’s launched go to the Storage tab.

Gibraltar Hub Administrator Storage Configuration

Hub Administrator - Storage

You’ll notice you have a few options - how long data is kept and how much data to keep.  Depending on the parameters you enter it’ll display an estimate of how much disk space you need to keep that log data.  Once you’ve made any changes you want click Save.

Your changes will be applied the next time the Hub Service runs its file prune process which it does roughly once an hour.  There’s no need to restart any services for this change to take effect.

That handles file storage, but if you’re going to have all of that log data you’re probably also keeping a lot of sessions around as well.  We recommend you upgrade from the embedded database to SQL Server for your data storage index.  You can use any SQL Server 2008 and later server, including SQL Express.  To switch from the embedded database to SQL Server, click the Change button next to the Index database.

Repository Index Selection

Repository Index Selection

You’ll need to enter enough information to connect to your SQL Server and specify a database that already exists.  The schema for the database will be fully populated by the Admin tool once it connects.  Before doing this, be sure to read through our documentation on using SQL Server with hub, it has a number of notes to cover pretty much any configuration situation.  If you have any questions about your specific situation, reach out to us through support and we’ll be sure you get taken care of before you start making configuration changes.

When you make this change you’ll need to restart both the Team Service (Windows Service) and the web site to make sure it takes effect.  Note that with the current release of Gibraltar there’s no data migration from one index database type to another so if you convert to SQL Server you will have a brand new repository.  You’ll want to make this decision early before you collect a lot of data!  The next release of Gibraltar will have a built-in migration to take care of this.

Related Posts

Loupe 4.5 Released with New Log Viewer for Web

Rapidly diagnose each error in any .NET application with our new Web Log Viewer and Exception root cause analysis, new in Loupe 4.5. New integration with Azure Service Bus and Azure Search enables full Loupe functionality without any Virtual Servers in Azure. Read more

Cloudflare Vulnerability Does Not Affect Us

The recently reported Cloudflare vulnerability where fragments of secure, encrypted user data could be exposed to a third party does not affect Gibraltar Software even though we use Cloudflare because we only route static content through the Cloudflare proxy for acceleration. Read more

We're out of our Last Data Center

Back in January of 2016 we decided to completely transition out of our data centers and into the cloud. On Sunday we finally shut down the last cluster of our hardware. Read more for how we did it and whether we would do it all over again if we had... Read more

Rock solid centralized .NET logging

Unlimited applications, unlimited errors, starting at $25/month