Using Data Archive in Microsoft Sentinel
An overview on how archiving data works in Microsoft Sentinel and how to restore old data.
There maybe different reasons organizations need to retain data longer than the 90 days that is normally configured for keeping data in Microsoft Sentinel. This could be due to regulations that require a company to keep records for a minimum of 1 year, or for internal investigations that could span back over the past year. There are many reasons why this functionality is needed and Microsoft Sentinel provides a very simple method of storing data for the length the organization needs.
Configuring Archive Retention
Within the log analytics workspace that Microsoft Sentinel is installed upon inside the tables section this will list all tables that can hold data and the data retention limits which can be set for each.
For example, if the table Azure Active directory Risky Users table would need to be retained for at least 1 year, the table could be configured to archive the data. The three dots at the far right of each table will give the option to manage this table. In this management window the “Total Retention Period” can be modified to push the logs into an Archive tier once the interactive retention has finished.
The screenshot above shows to keep a years logs 90 days of logs will be kept in interactive retention state after this 275 days will be kept in an archive retention state. This allows users to keep logs for longer periods at a reduced cost. Current costs to keep logs in this archive state are shown below:
(Source: https://azure.microsoft.com/en-gb/pricing/details/monitor/)
Restoring Archive Data
If there is a reason archive logs will need to be restored there are a couple of steps that can be used to minimize the amount of data that needs to be restored. If there is an incident or a specific use case that requires restored data, the “Search” function within Sentinel should be used first. As not all data may be required to be restored the Microsoft Search function in Sentinel can be used to check the archive logs has the information needed and can be used to confirm the time frame that data is needed to be restored from.
As there are costs to restore data from the Archive retention state back into an interactive retention state, the Search function is significantly cheaper to check the archive data before the commitment is made to restore.
(Source: https://azure.microsoft.com/en-gb/pricing/details/monitor/)
To run a search job open Microsoft Sentinel and select the Search section, your screen should look something similar to below:
Within the search bar you can enter a keyword such as a users account that you may be investigating or you could add nothing to not limit the search job at all over the table. You can select a specific table under the filters section. A logs window will appear to show to KQL used and this is where you are able to change the time range to go back to a specific time window where you believe the logs required to be restored will be.
If any log matches the query, a limited summary of the data will appear below the query, this is where any changes to the time range can be made before configuring the restore.
Once happy with the time range, under the Search Section, the Restore button at the top can be pressed to configure the restore from the specific table during a specific time windows:
As detailed in the restore section there are some restrictions to the restore functionality that should be noted:
Restoration limits
Minimum of 2 days of log data to restore.
Upper limit of 60TB per single restore
Up to 4 restores per table per week.
Up to 2 restore processes in a workspace can be concurrently running.
One active restore at a time per table.
When restoring data from a table the following costs for doing so are detailed below:
(Source: https://azure.microsoft.com/en-gb/pricing/details/monitor/)
To help understand how much the restore may cost, the Microsoft Pricing Calculator can be used to estimate the costs. The amount of GB that would be restored will need to be known to utilize the calculator effectively.
https://azure.microsoft.com/en-gb/pricing/calculator/?cdn=disable
Once the restoration has completed, under the tab “Restoration" you should see the restore and if the data is available, you will be able to click on the table and a logs section will open showing the data which can be queried.
(Source: https://learn.microsoft.com/en-us/azure/sentinel/restore )
Once the data is no longer needed it is imperative to remove this data as the daily cost of keeping the logs in the interactive retention will be costing standard amount per GB per day:
Removing the archive data is very simple and again can be done from the restoration section, on the right of any data restores there is a delete button that can remove the data.
(Source: https://learn.microsoft.com/en-us/azure/sentinel/restore )