Gibraltar 2.5 New Feature Dive - Email Notification

Quick Takeaway

With Gibraltar 2.5 we’ve added email notifications to the Hub server.  You can receive text or HTML-formatted messages when new sessions arrive at the hub or detailed emails when errors are reported.  You can read about it in the online documentation.   The messages are pretty and we’ve worked hard to make sure you won’t get flooded when something goes wrong.

The Full Story

With Gibraltar 2.5 we’ve introduced a new server-based email notification system that can be easily configured by your team.  Our original goal was to satisfy a common Hub user request to give them some way of knowing that there was new data worth looking at so they’d know when to power up Analyst, retrieve updates and dig in.  As we scoped out the mission of Gibraltar 2.5 we saw another essential characteristic:  Email is the most pervasive process management tool in the world.  By providing the right email messages from the server, we could integrate with the defacto processes of a large number of customers.

• About a third of Gibraltar customers don’t have a dedicated issue tracking system.  They informally use email and spreadsheets to track what’s important.
• Most issue tracking systems can intake issues via email, so email notifications become a default integration option for these systems.

As we went through the design process for notifications, the key mission we kept laying out is that they had to be good enough to let you know if you needed to leave a meeting with your boss to go dig into an issue.  Think about it:  You’re sitting there in a meeting and your phone vibrates with an email,  You steal a quick glance and need to see enough to decide no biggie, it can wait or uh oh - better make my apologies and get back to my desk.  This meant it had to be readable on mobile mail clients (iPhone, Android, etc.) as well as your full email client (Outlook, GMail, etc.).

What Kind of Notifications?

We decided there needed to be two types of notifications:

1. New Session Notification: Alerts you that an interesting application session has been received with a summary of the overall session.
2. New Error / Warning Notification: Alerts you that an error or warning has been received.  You get an email for each unique error or warning in a session.

Sometimes you just need to know if there’s new data available and if so a very quick summary of what it contains.  This is primarily interesting for users that don’t receive a lot of sessions, for example where sessions are only submitted when an end-customer has a runtime problem with the application or you’re primarily supporting one or two web sites.

For this scenario you can select Send a Session Available Notice and get few emails that are fairly high level.  You can see from the example that you get a summary of the session including what exact application & version it related to and the most commonly used information about the computer where it was running.  After that, each unique message that matched your notification threshold is summarized.  If it occurs many times the total number of occurrences is listed along with the timestamp of the first occurrence.  This format is designed to handle scenarios where a single error (like being unable to open a database connection) results in a storm of other errors (e.g. record not found) you can see both instead of the first error being lost in the storm.

The other scenario is to see deeper drill-in information on each specific error as they happen.  This is needed if you want to integrate with an external issue tracking system (since it needs something specific and somewhat unique to build an issue from) or you need to be able to take action on a problem quickly, possibly without even starting up Gibraltar Analyst.

For this scenario you can select Send Details on Qualifying Messages and get an email for each error individually.  The full details of the log message will be written out in an HTML format optimized for quick visual scanning.

If the log message that triggered the notification includes exception details the exception(s) will be written out to the detail with the call stack for each in order from outermost to innermost.

At the end of the message  a session summary is included with the most common information about the application & computer to help you perform quick triage.

Safety first - Notification without Noise

One thing we were worried about from the start was flooding prevention. You don’t want to be bombarded with 1300 emails because a web server went crazy and started screaming about a connection problem or your server got hit by a DoS attack.   With session notifications this is easy because you’ll just get one notification per session which significantly reduces the rate they can happen.  With log messages it’s a different story.

What we did was leverage an existing idea built into Gibraltar of grouping messages by a fingerprint - typically the class & caption or the category & caption.   In our experience this tends to be a fair grouping of “sameness” - The captions tend to only include a modest amount of really variable content (like file names & user names) but are unique enough particularly when paired with the location the event happened.  By grouping messages in this way recurring errors that might otherwise flood you are all pulled together and carried as a single alert.  We display the first occurrence in full detail then just let you know how many more times it happened in the same session.

Getting Just the Alerts You Want

It was clear to us from the start that notification couldn’t be as simple as “always send an email when there’s an error”, it was going to have to be more contextually sensitive or it’d be turned off.  We’re highly opposed to features that look great in a demo but not in the real world (e.g. demoware) so we knew it had to be easy to configure to just “do the right thing”.  We also work with enough real world customers to know that it had to be configurable from Analyst.  In many shops the Hub server is run by IT who treats every change to its configuration as a change request.  That barrier, no matter how well meaning, would create enough friction to make notifications less nimble in the real world.

To configure notifications, you just select Notification Options from within Analyst.  As long as you have a Hub connection you can then view & edit all of the notifications and changes will take effect on the server as soon as you click OK.

We wanted the notification rules to be as simple to understand as possible, but we wanted to hit all of the reasonable user scenarios we could.  The basic model used for a notification rule is to define the matching sessions, indicate what type of notification you want, and then where you want it sent. You can pick a single entry from the address book which in turn forwards to a single email address. If you want to sent it to a group of people, just put in the address of the right email distribution list. The address book makes it easy to update a set of destinations in one place, as well as disable HTML formatting for emails if necessary. For example you could create a single address book entry for your issue tracking system with HTML turned off and then one for yourself with HTML turned on.

Finding the Right Sessions

You can match sessions by specifying just the required elements:  Product, Product & Application, Product & Application & Version and separately Computer.  For Version you can match exactly, less than, or greater.

We considered more elaborate version matching capabilities but as we walked through user scenarios we just couldn’t justify the added complexity of the user experience.  In a typical environment either:

1. Version is irrelevant: If you have a web site and aren’t tracking version, then it doesn’t matter at all.  In this case Computer is more important to separate out environments.
2. Production & Development Versions: If you are tracking versions there are usually older version(s) that are in production and then after a certain point all newer versions are development/QA/experimental.  You then are looking to either match production versions or development versions which just requires less than and greater than.

We knew from our customer conversations that some folks were very concerned about warnings, and most were just concerned about errors and crash events (critical messages in Gibraltar).  You can configure how bad something has to be to qualify in the same editor.

The Server that Cried Wolf

There are often computers that you never want to hear from - typically development systems or internal test systems.  You want to be able to run the production code in the production configuration - so they’ll still submit sessions to your Hub - but not want to be bothered when there’s a problem.

To solve this, you can add the computer to the list of suppression rules and that will prevent any related session from generating a notification even if it matches a rule.  This simple approach will get rid of most of the noise alerts.

Avoiding Duplicates

When the server is processing a session it calculates the messages it needs to send based on all of the rules and then sends each unique message to the set of qualifying recipients.  This effectively merges all of the rules together that would cause the same message to be generated.  It then eliminates duplicate destinations by email address (so if two address book entries use the same email address and both qualify for the message only one entry is added).  Finally, since a single message with all of the addresses is submitted to the mail server the mail server can eliminate duplicates once it expands address lists.

All of this is designed to prevent you from receiving multiple copies of the same notification even if you are obliquely included due to distribution lists.

Another way you may get duplicate messages is due to the same session being provided to the hub multiple times.  This can happen for a few reasons, most commonly because it’s being submitted incrementally as problems are happening but the session hasn’t ended yet.

To prevent the notification engine from reprocessing messages it has already examined the Hub server keeps track of exactly what set of messages have previously been checked and ignores them when the session is resubmitted.  This way only new messages are processed.  If you read the resulting alerts carefully you can see when this happens because the notification subject will say that an updated session is available or the error detail will describe “new” occurrences of an error.

Broad Compatibility

Since notifications are primarily about getting information out to other systems and for use in situations when you may not be able to run Analyst it’s more important that they look OK wherever you are than that they look awesome in any particular viewer.  If you’ve ever worked on HTML formatting an email you know the differences between email clients are stunning (and really appalling - You’ll be partying like it’s 1999).  We did a range of testing of each of our HTML formats using:

• Outlook 2003, 2007, 2010 (It’s what we use along with most of our customers)
• Outlook Web Access 2007, 2010 (Also what we use)
• Gmail (second most popular mail client our customers use)
• iPhone Mail

Just between the first three we had major problems to work around - each ignored font sizing or coloring when declared in different ways for example.  Ultimately we found a combination that worked reasonably in all of them.

When we generate each email the HTML version is technically an attachment to the main text version.  This way a client that doesn’t understand HTML formatting will see the text-optimized version and continue unaware of what it’s not seeing.

Just the Text Please

If you are sending emails to an address you know you never want the HTML version for (either because it just doesn’t display well or it just isn’t necessary because you always want it as text) you can disable the HTML version from being generated & attached by clearing the HTML Formatting option in the address book.  The Text versions do more-or-less fixed width text formatting to attempt to create tables of items where necessary and may be parseable if you are feeling frisky.

Get Notifications Today

If you’re using the Gibraltar Hub Service, just download & install the latest version of Gibraltar Analyst and you can start receiving notifications today.  You don’t need to do anything to your deployed application - this change only affects the Hub & Analyst.

If you’re using your own private Hub you’ll need to first upgrade it to the latest Gibraltar Hub before you can configure notifications.  You’ll also need to configure a mail server for your private hub to send mail through - complete details are in the user documentation.  If you or your IT department have any questions just contact our support department via email, we’re here to help.