Handling Hundreds of GB of Log Data with Hub

Behind door 6 of our Advent series we have today’s walkthrough on how you can handle massive amounts of log data with Hub:

When you first start out with Gibraltar you usually don’t generate a lot of data - our data files are highly compressed, your sessions in development aren’t too long and there aren’t too many of you.  Then comes the day you launch everything into the wild where people might run your application for days, weeks, months…  Perhaps you have many thousands of users.  Now the data is really scaling up!

Fortunately, the Hub is designed to handle this.  Out of the box the hub operates with a pretty conservative data limit - 2GB.  Once you have more than that amount of log data the oldest data (based on when it was received) is deleted until it’s back within the storage limits you set.  But, say you’re running a private hub and have a few hundred GB of free disk space: let’s put that to work!

This change has to be done directly on the Hub server, so log into the OS on your Hub Server and start up the Gibraltar Hub Administration tool.  Once that’s launched go to the Storage tab.

You’ll notice you have a few options - how long data is kept and how much data to keep.  Depending on the parameters you enter it’ll display an estimate of how much disk space you need to keep that log data.  Once you’ve made any changes you want click Save.

Your changes will be applied the next time the Hub Service runs its file prune process which it does roughly once an hour.  There’s no need to restart any services for this change to take effect.

That handles file storage, but if you’re going to have all of that log data you’re probably also keeping a lot of sessions around as well.  We recommend you upgrade from the embedded database to SQL Server for your data storage index.  You can use any SQL Server 2008 and later server, including SQL Express.  To switch from the embedded database to SQL Server, click the Change button next to the Index database.

You’ll need to enter enough information to connect to your SQL Server and specify a database that already exists.  The schema for the database will be fully populated by the Admin tool once it connects.  Before doing this, be sure to read through our documentation on using SQL Server with hub, it has a number of notes to cover pretty much any configuration situation.  If you have any questions about your specific situation, reach out to us through support and we’ll be sure you get taken care of before you start making configuration changes.

When you make this change you’ll need to restart both the Team Service (Windows Service) and the web site to make sure it takes effect.  Note that with the current release of Gibraltar there’s no data migration from one index database type to another so if you convert to SQL Server you will have a brand new repository.  You’ll want to make this decision early before you collect a lot of data!  The next release of Gibraltar will have a built-in migration to take care of this.