Reworking the Gibraltar AddIn API

There’s a classic saying in designing software that you should plan on throwing the first one away.  I’ve never been very comfortable with that because it can be easily abused by a team to justify half-baked thinking, inadequate requirements and design, and just general hope instead of solid engineering.

That said, it’s generally been my experience that no matter how good and diligent you are when designing an API you’ll inevitably find that you’ve missed the mark as soon as the first set of users start solving real problems with it.   So, you’ll want to leave room to redo and rethink your original approach before you’re married to it.  This is the major reason that we shipped Gibraltar 2.2.0 as a Beta back in early April with our first vision of a Gibraltar Add In API.

Feedback from inside & outside

To make sure we got solid evidence that the design was sound we worked hard to get some existing customers as well as outsiders to try solving their challenges with the API.  We also did our own internal Red Team approach:  we asked Jay (who isn’t doing any day-to-day development on Gibraltar, but has extensive commercial development experience) to scratch his favorite itch with the API as well.  We’re going to publish more about what he built soon, but without going into details something significant emerged from his work and discussions with our customers.

Instead of people using the API to solve a few tactical problems (like data export or an alternate view of the data) people wanted to use it to extensively integrate Gibraltar with external systems. This went way beyond our original intent; we have a big block of features on our roadmap for creating an external integration engine and hadn’t ever intended this to be the way that came about.  Despite that, it’s the problem people kept wanting to solve with the API.

Unfortunately, there are several things that we just didn’t allow for, but are common problems in these scenarios:

• Configuration Management: Whenever you’re working between two systems, you’ll want some metadata to describe endpoints and map data between the systems.
• Credential Management: Nobody likes hardcoding credentials, or doing something simple like popping them in a file without encryption.  We all know the right things to do, but it’s annoying and overhead.  It should be built in.
• Coordination: All the various user interface and process extensions don’t exist in a vacuum; they need easy ways to coordinate their work to enable new processes, not just isolated features.
• More Extensibility: People want to be able to extend other parts of the UI again with the goal of connecting processes together.
• Development Debugging: In a twist of irony, it’s hard to log and view data from within an Add in because you’re running within the Gibraltar environment; you can’t use the Agent to log because you’re running inside of the Agent.  This made development of extensions a pain.

Powerful capabilities, not demoware

As we’ve said before, we hate demoware.   Although we’ve heard from a few parties that we just need to “check the feature box” with the API (have it but no one will use it), we just aren’t willing to accept that measure for success.  Instead, we see another angle:  We’re going to leverage the add in API ourself to enable us to create targeted features that only apply to some part of our audience so we can create them faster and keep control over the product’s complexity.  In short, we’ll make sure that it’ll solve problems for you by having it solve problems for us.

Fortunately, we didn’t have to throw the first one away in a literal sense.  We did end up discarding virtually our entire implementation of it from the Gibraltar side, but the change to the API you see is fairly modest.

We’re still finalizing the details, and deciding whether we need to do another beta round of 2.2 or should go ahead and formally release 2.2 so that everyone can get the benefit of all of the improvements that have happened over the past several months beyond the API.  But, we can speak to a few great features:

Configuration Management

Configuration data can be shared between the Hub and all of its connected Analysts.  This makes it more practical for add ins that connect systems together but need everyone on the team to share metadata on how items in both systems relate.  Since add ins can run on both the Hub and in the Analyst we made it so you can change the Hub configuration for an add in remotely.  This means you don’t have to run to IT and get on the server to say add new mappings for new projects or do whatever other configuration your favorite add in requires.  The central data is automatically synchronized between the Hub and Analyst and is available even if the Hub isn’t (say because you’re running disconnected from the network)

For example, say you want to integrate with your defect tracking system.  It’ll have some form of heirarchy for managing things - probably with different areas for the current development release vs. the production releases, etc.  You need to be able to relate the data Gibraltar has (like product and application names as well as versions) to those areas so when it opens or updates an issue it shows up in the right place and in the right workflow.

Credential Management

In a perfect world, your Windows credentials would give you access to any external system your Add In needs to access, so no credentials need to be stored.  Unfortunately, most of the third party systems you’ll want to integrate with (particularly ones that aren’t designed for enterprises) just flat won’t integrate that way.  You need to be able to store a user id and password but do so in a way that doesn’t create an unreasonable security risk.

We’ve baked in credential management leveraging the data protection API in Windows (DPAPI) so that you can encrypt account names and passwords so they can be accessed only by the current user, computer, or the hub.  Here again you can manage it all from Analyst, so if you need to change the account the Hub is using to login into your defect tracking system or something you don’t have to go to the server to do it.

Development Debugging

We’re taking advantage of improvements in the Live Log Viewer control to offer a real-time view of what’s happening with Add Ins, both the Gibraltar side and the messages logged by each Add In.  We log a lot of information as Analyst runs, and we don’t want to bury you in all of that detail so it automatically filters down to just information needed to monitor and support Add Ins.

This is done using only capabilities that are exposed through our normal API, so you can do the same thing in your WinForms/WPF application as well.  You’ll notice we’ve updated the live viewer to include a number of capabilities formerly only available in Analyst, like source code lookup and exception / detail viewing.

More Extensibility

We’ve added a number of places you can extend Gibraltar to the original list:

• Analyze Sessions on Hub: Any analysis engine you create can run in Analyst or Hub.  We always intended it, but it wasn’t ready for the beta.  This means you can have data flowing from Gibraltar to external systems without anyone being logged in or having to leave something running.
• Global Commands: You can extend the main Analyst menu bar with your own menu items to create global commands end users can easily access.
• Controllers: We’ve added some key controller interfaces so you can hold state in the right places (think MVC) and substantially improve the efficiency of integrating with remote systems that may be expensive to query.
• And More: Frankly, we’re going to continue adding items as we head towards release based on our internal experimentation of what we can safely get done and commit to.  Our ultimate goal is that you can extend each of the different views in Analyst so you can have it seamlessly participate in end-user workflow.

Real World Experience

We’ve been internally using the new API to help us automate our CEIP process for Gibraltar itself and it’s given us a lot of confidence not just in the new implementation but also in the vision it represents.  We’re going to be able to ship extensions outside of our release cycle, add optional modules in the box for a great initial experience, and help create a space for you to show your peers the cool things you can do where Gibraltar + Your System  = Awesome.

Is there a specific system you’d like to see us integrate with?  Drop us a line; we love to engage with our peers to understand new & creative ways to make a difference in how developers create rock solid .NET software.