View Massive Logs, New in Loupe 3.6
Victims of our own Success
It happens time and again - work hard to address scalability or performance in one aspect and it’ll show up in another part of the system. With Loupe 3.0 we made it dramatically easier for people with long running or very large sessions to reliably get that log data all the way down to Loupe Desktop from remote systems. The results? We routinely see customers with multi-million log message sessions, sessions that lasted over a year, and other extremes we didn’t envision when we were creating Loupe Desktop.
For a variety of internal technical reasons, Loupe Desktop opens an entire session into memory to display it - so if you double-click a one million log message, it’s going to load and display the whole thing. We did a range of tricks early on to reduce memory footprint with the goal of this being reasonable up to around one million messages on the large side. But, when you have a 40 million message session there’s no strategy that begins “load the entire thing into memory” that’s going to end well.
How do you load 40 Million Messages? You don’t!
For Loupe 3.6 we’ve addressed this scenario by adding a range selection dialog that displays automatically when it detects the log is large enough that it’d take a significant amount of memory or time to load.
The selection dialog shows a histogram of the log data including the log message rate (the blue bars) and the error rate (the red bars). Adjust the selector to the time range you want and it will estimate how much memory is required to load that part of the session. If you may not have enough memory to load the session it’ll indicate in red as well
Once you click OK, just that part of the session is loaded. By default we pick the tail of the log which usually is where people are interested in starting so if you just click OK without thinking you’ll get a useful block of data.
If you need to see more of the session or a different part after you’ve displayed it, you can just close the session and re-0pen it to select a different timeframe.
Under the Hood Improvements
We’ve also optimized the download process and session open processes - Downloading now breaks work into multiple threads to run faster over high bandwidth connections (saving up to 50% of the time it used to take to download a large session) and made dramatic improvements to the time it take to open a session with a large number of fragments.
When we originally designed Loupe 3.0 we assumed a 90th percentile session would have 1 million messages and as many as 20 session fragment files. In the real world we’ve seen as high as over 4 billion messages and 100,000 fragment files. Opening a billion message session isn’t just a function of needing to pick a range, you have to work with all the fragment file quickly as well. We optimized this code path and reduced the overhead for a large number of fragments to at most a few seconds of work when opening the session.
Go ahead, find the next limit!
This is all in the service of the original goal of Loupe - log as much as you want and don’t worry about it, we’ll make the infrastructure that can handle it. After all, you can’t get in a time machine and go back to yesterday to turn on verbose logging right before the system went down or that critical customer error happened. You need to be running with enough detail all the time you can go back and resolve issues fast!
Take advantage of unlimited logging today - download Loupe Desktop (it’s free!) and go wild.