When performing OMS log search programmatically, you will encounter an API limitation that will prevent you from getting all the logs from the result set. Currently, if the search does not include an aggregation command, the API call will return maxium 5000 records. This limitation applies to both the OMS PowerShell module (AzureRM.OperationalInsights) and searching directly via the Log Search API.
The return response you get from either the Get-AzureRmOperationalInsightsSearchResults cmdlet or the Log Search API, you will get the total number of logs contained in the result set from the response metadata (as shown below), but you will only able to receive up to 5000 records. Natively, there is no way to receive anything over the first 5000 records from a single request.
Last month, I was working on a solution where I needed to retrieve all results from search queries, so I reached out to the OMS product group and other CDM MVPs. My buddy and the fellow co-author of the Inside OMS book Stanislav Zhelyazkov provided a work around. Basically, the work around is to use the “skip” command in subsequent request calls until you have retrieved everything. For example, if you want to retrieve all agent heartbeat events using query “Type=Heartbeat”, you could perform multiple queries until you have retrieved all the log entries as shown below:
- 1st Query: “Type=Heartbeat | Top 5000”
- 2nd Query: “Type=Heartbeat | Skip 10000 | Top 5000”
- 3rd Query: “Type=Heartbeat | Skip 15000 | Top 5000”
- … repeat until the search API call returns no results
I have written a sample script using the OMS PowerShell module to demonstrate how to use the “skip” command in subsequent queries. The sample script is listed below:
Here’s the script output based on my lab environment: