Reducing the Storage Demands of .NET Core Log Files
Despite TB hard drives becoming pedestrian, handling storage is still a struggle for many. Data caps, expanding file sizes, and more put pressure on software developers to cut down the storage demands of their applications, both for their customers and themselves. Often, one of the first places a team will look is at their logs. Why are they so big? Why are there so many? And do you really need those log files from 3 years ago?
In this article, we will go over what makes log files large and ways to reduce the pressure they put on storage. It’s tempting to delete them early and often or even stop logging. But if we think about how we can affect storage size at every stage of a logs lifecycle, we can keep our logs for a long time and ensure we have the data we need.
Why are Logs So Big?
The simple answer is logs are generally text-based files, and large logs contain many characters. Each character is generally a byte, which adds up once a log file holds thousands of messages.
But it’s a bit more complicated than the number of messages, as each message can vary in length dramatically. From variable and method names to the inclusion of diagnostic details, these all can add significant size to each message. So detail-rich logging (the kind we use and recommend) will take up more space than bare-bones messages.
However, we don’t recommend removing diagnostic details from your logs or changing your program to use method and variable names that are short and undescriptive. Logs are supposed to tell you what’s going on with your application under the hood, and they need to have enough detail that you and other developers can reliably interpret them. So when considering the size of your log files, we need to start by using strategies that help us prioritize the vital information while retaining usability.
How Can I Reduce My Log File’s Size?
We aim to have detailed logging that tells the story of an application session without any filler and takes up minimal disk size. To do this, we will filter out less important log messages and then compress the log file once written.
Optimizing What Logs are Written to a File
The easiest way to optimize what logs are written to the file is by filtering with log severity. For .NET Core, logs will have one of the following severity levels:
- Trace
- Debug
- Information
- Warning
- Error
- Critical
In almost all circumstances, you can filter out Trace and Debug severity logs. They contain an excruciating level of detail, and I only use them when actively troubleshooting with a live log viewer. In most other cases, these logs will clutter up the file without providing a better understanding of what actually happened.
With .NET Core, you can control which severity level you write logs for using the Logging section of appsettings.json file:
"Logging": {
"LogLevel": {
"Default": "Information"
}
}
This setting will allow the active loggers to write Information severity logs or higher. If you wanted to reduce your log amount further, you could change the Default setting to Warning.
I personally find most information level logs valuable for providing context for the higher severity messages. Still, you may want to set the severity level to Warning for specific categories of logs, like those created from third-party extensions. You can do that by specifying the category name and setting the level to None:
"Logging": {
"LogLevel": {
"Default": "Information"
"CategoryXYZ": "None"
}
}
Microsoft’s documentation covers these settings in detail if you want to learn more. This may not be the only way you want to modify your logging output, but it’s an effortless way to start that’s quickly reversible.
Optimizing the Log File For Storage
Compression is the primary way to reduce a file’s size for storage. Specifically, minifying the file/zipping the log files can save you some space for storage. If you are primarily logging to standard text files, you can simply zip anything you’re not actively working with and decompress as needed.
For Loupe, we write the logs to a binary format instead of human-readable text. This is because the file is ultimately read through Loupe, so we rely on Loupe Desktop and Loupe Server to convert the file to human-readable text and display it nicely.
The result is a log that takes up about one-third of the storage of a regular text file (potentially one-fifth of the storage) and still less than a minified or zipped file. The main downside of this approach is you can’t read the file using a text editor. But Loupe Desktop is completely free, and Loupe Server can run on any machine with a browser, so the logs are still very accessible.
How Long Should I Retain Logs?
It’s a good practice to retain logs for around a month if possible. This provides an easy cutting-off point to follow when cleaning and is generally enough to work with when investigating bugs.
Some developers rely more on tracking a set amount of storage than a time cap. They may limit their log data to a value like 100MB and replace data as the cap is hit. This has appeal as a “set and forget” option but prone to issues if the application receives increased usage or produces an error thousands of times. Suddenly, the log file may only contain a few days/minutes of logs, which is far less helpful when solving a complex problem.
There is merit to using a set amount of storage for logging, but only when that storage is large enough to retain a month or more of log data. If your application has predictable logging, feel free to set a cap. But be prepared to adjust that amount with increased usage or when an error gets wordy.
With Loupe Server, we generally retain the log files themselves for a month but retain the session metadata for a year. This allows users to get the most bang for their buck while retaining a documented history of their application usage. You can adjust the retention period for both items on request.
You can set a log retention period for Loupe Desktop or keep everything. I keep all the logs as it doesn’t take up that much storage. Our compressed binary logs are small enough that it hasn’t been an issue.
Looking to Reduce Your Log Storage Concerns?
If you have storage concerns with your logs, it’s worth giving Loupe a try. You can try Loupe Desktop for free or learn about the Loupe Server Trial in the link below.