Scale Out with Enterprise Edition, New in Loupe 3.6

This is the third in a series of posts on new features in Loupe 3.6

When we created Loupe Server it always supported two modes of operation - a single tenant solution which was available to be installed by anyone (Loupe Server) and the Software-as-a-Service multitenant system which we operated for you (Loupe Service).  Our main reason for building multi-tenancy into the code was to reduce our hosting costs so we could offer inexpensive options and healthy trials without having to create a kajillion installations. We hadn’t expected anyone else would really need these capabilities - we focused on ensuring Loupe Server scaled up to large data volumes and had internal data segregation to meet most needs.  This worked for a while, but after Loupe 3.5 shipped we started getting customer requests to have some or all of the features unique to our SaaS version deployed within their infrastructure.  Working with these folks we saw a few common requests:

1. Load Balanced Clustering: Some shops just require everything to be active/active redundant.  Others only have certain stock VM configurations so scaling “up” isn’t supportable in their environment.  Regardless, they wanted to have two, four, or more servers collaborating to handle their load.
2. Strict Data Segregation: Certain companies have regulatory or procedural reasons they want to create hard partitions between the log data from different environments, applications, or teams but want to share common infrastructure to lower operating costs.
3. Separation of Roles: Typically for network security reasons a company may want to have distinct endpoints for receiving data, using the Web UI, and data access.  Background processing servers may be separated from these for performance or network access reasons.

These capabilities were already present in our SaaS since we needed them for Loupe Service.  What was missing were suitable administration tools and some detailed work to meet larger customer expectations on maintenance processes.  While working through the beta project for Enterprise we unearthed a few new requirements that our SaaS could benefit from that we might not have taken on just for our own needs.

Segregating Data - Yours, Mine, and Ours

Often, different teams within an organization will want different repositories so they have complete independence to choose details like global notification rules, event handling rules, release types, and other workflow elements.  While they want this independence, their organizations want to operate one set of common, high performance infrastructure.  Individuals may be members of multiple teams so they should be able to access the data for every team they’re a part of. Enterprise addresses this by letting you create an unlimited number of Repositories.  A repository is just like a Tenant in the Loupe Service - so most actions happen on a single repository.  When Agents send in data, they send it to a repository.  Each repository has its own list of users allowed to access it and their role for that repository.   The set of users is common to the entire installation, so if someone in your company has access to your Loupe Server then you can add them to one or more repositories and they can move between them in the same session (without logging in again or having separate accounts). Each repository gets its own SQL Database, file folders, and search index: no data is shared between repositories.  This works for organizations that may require strict segregation of where data is stored for regulatory reasons.  Data retention rules are also per-repository so each team can make their own decisions about how much data to keep and for how long.

Scale Out with Clustering

There are several scenarios you might have which are addressed by clustering in Enterprise:

• Redundancy: You want to be sure regardless of whether a VM is taken out you’ll still be up and capturing information.
• Separation of Work: You want to separate the high volume data API and the background task processing, typically because you have your own custom Loupe AddIn that is doing extra work and may even need to be in a different network.
• Scalability on Commodity Systems: You may want to handle a large volume of sessions and data using virtual machines with 2GB of RAM and one core instead of scaling up a single system.

Enterprise lets you split your Loupe environment into different roles for the Web UI, API Transport (where agents send their data) and Analysis (all background processing).  Each of these roles can be clustered with any number of servers.  To cluster the web UI and API you’ll need some form of external load balancer but Loupe makes no requirements on it for sticky session or other advanced configuration steps. Behind these servers, data is stored in SQL Server and on the file system. You can employ multiple SQL Servers and multiple file servers (or SAN/NAS systems) to share the load as needed, provided they are all visible to all the Loupe servers in your environment.

Terabytes of Data at your Fingertips

Lets say you originally went to your storage team and got a 120GB disk for log storage.  Down the road you decide you want to keep 512GB.  The storage team can give you a new 512GB volume but they can’t change the original’s size.  Or, you want to store 5 TB of data but your storage team will only allocate volumes 1TB at a time.  These aren’t hypothetical examples - these were real customer situations when we originally gave them Enterprise. Talking with customers, we were clear that provisioning storage is something that needs to be flexible and easy in Enterprise:

• All of your data may not fit on a single volume.
• Adding and removing volumes couldn’t require taking the system down.
• Storage may be pooled for several repositories or a single repository may get its own dedicated storage, depending on the organization’s rules and requirements.

To handle this, Enterprise has Storage Pools.  With these you can define multiple storage volumes and Loupe will load balance new data across them.  You can even pick the load balancing approach - do you want it to fill the available volumes in order (minimizing the number of volumes used) or distribute the data across them for best performance? This system improves availability too - Loupe probes each volume in the storage pool to confirm it’s accessible and has free space.  When a volume fills up or isn’t accessible it will be skipped until it comes back online.  Sure, any data previously written to that volume won’t be available but at least new data will be! You can age out old storage by changing the volume to be either read only (no new data will be written to it) so as data is purged it will eventually be emptied.  So if you need new storage you can create a new volume, mark the old one as read only and just wait for the old data to age out and then remove that volume completely.  All of this can be done without Loupe ever missing a beat.

What’s this like in the Real World?

We’ve been using this software for a long time as our Loupe Service and completed the new features for Enterprise several months ago, at which point we’ve been running them on our extended SaaS infrastructure which processes multiple terabytes of data per day across 9 virtual servers.  We’ve chosen to separate all of the roles - API access, web access, and background processing - onto separate servers to optimize web response time.  (this is why you use Loupe.GibraltarSoftware.com </a>to access the web app but your agents and Loupe Desktop connect to Hub.GibraltarSoftware.com). We have several corporate customers also using Enterprise edition - we gave them a special internal version (Loupe 3.5.7) which had the new administration tools and improvements.  They helped us identify a few new use cases that needed administration tooling changes, documentation, and some other details.

How do I get it?

Loupe Server is now offered in Standard or Enterprise Edition.  The only difference is the server licensing - users are the same for either.  If you already have Standard and want to upgrade email sales@gibraltarsoftware.com and we’ll credit you your existing license to upgrade.  Otherwise, we recommend organizations try out Standard edition to determine if Loupe Server is right for you and if so go ahead and purchase Enterprise.  Every Gibraltar Software purchase comes with our 45-day satisfaction guarantee so you’re covered!