Handling Hundreds of GB of Log Data with Hub

Behind door 6 of our Advent series we have today’s walkthrough on how you can handle massive amounts of log data with Hub:

When you first start out with Gibraltar you usually don’t generate a lot of data - our data files are highly compressed, your sessions in development aren’t too long and there aren’t too many of you.  Then comes the day you launch everything into the wild where people might run your application for days, weeks, months…  Perhaps you have many thousands of users.  Now the data is really scaling up!

Fortunately, the Hub is designed to handle this.  Out of the box the hub operates with a pretty conservative data limit - 2GB.  Once you have more than that amount of log data the oldest data (based on when it was received) is deleted until it’s back within the storage limits you set.  But, say you’re running a private hub and have a few hundred GB of free disk space: let’s put that to work!

This change has to be done directly on the Hub server, so log into the OS on your Hub Server and start up the Gibraltar Hub Administration tool.  Once that’s launched go to the Storage tab.

Gibraltar Hub Administrator Storage Configuration

Hub Administrator - Storage

You’ll notice you have a few options - how long data is kept and how much data to keep.  Depending on the parameters you enter it’ll display an estimate of how much disk space you need to keep that log data.  Once you’ve made any changes you want click Save.

Your changes will be applied the next time the Hub Service runs its file prune process which it does roughly once an hour.  There’s no need to restart any services for this change to take effect.

That handles file storage, but if you’re going to have all of that log data you’re probably also keeping a lot of sessions around as well.  We recommend you upgrade from the embedded database to SQL Server for your data storage index.  You can use any SQL Server 2008 and later server, including SQL Express.  To switch from the embedded database to SQL Server, click the Change button next to the Index database.

Repository Index Selection

Repository Index Selection

You’ll need to enter enough information to connect to your SQL Server and specify a database that already exists.  The schema for the database will be fully populated by the Admin tool once it connects.  Before doing this, be sure to read through our documentation on using SQL Server with hub, it has a number of notes to cover pretty much any configuration situation.  If you have any questions about your specific situation, reach out to us through support and we’ll be sure you get taken care of before you start making configuration changes.

When you make this change you’ll need to restart both the Team Service (Windows Service) and the web site to make sure it takes effect.  Note that with the current release of Gibraltar there’s no data migration from one index database type to another so if you convert to SQL Server you will have a brand new repository.  You’ll want to make this decision early before you collect a lot of data!  The next release of Gibraltar will have a built-in migration to take care of this.

Related Posts

PostSharp Diagnostics Now Supports Loupe in the Box

The latest update to PostSharp Diagnostics adds Loupe support, enabling extensive high-performance logging to be added to any .NET application with virtually no code changes. PostSharp even has a great free option for developers that complements Loupe Desktop! Read more

Loupe Agent for .NET Core Now Available

The first release of the Loupe Agent for .NET Core is also our first open source version of the Loupe Agent. This is the first step in our plan to open source the entire Loupe Agent to make it easier for anyone to extend and take advantage of what Loupe... Read more

We've Moved Loupe Service to App.OnLoupe.Com

Loupe Service now has a shorter, direct site name that's faster, anywhere in the world. Just to go App.OnLoupe.Com, the new CDN-accelerated endpoint for the Loupe Service. Your existing Agents and Loupe Desktops are unaffected by this change, but access to the web UI will be redirected to the new... Read more

Rock solid centralized .NET logging

Unlimited applications, unlimited errors, scalable from solo startup to enterprise.