Welcome to part 4 of this series of blog post about LogParser and apologies for the long gap since part 3, here we are are going to look at continuous data extraction. Previously we have covered starting with LogParser in part 1, how to query and filter data in part 2 and in part 3 we reviewed how to collect data from multiple locations in parallel and how to get that data into a database.

One really neat feature of LogParser is its ability to keep track of the files its importing and skip rows that have already been scanned and only return rows added to the source since the last time it ran. We use a checkpoint file for this and we activate it in a LogParser query with the -icheckpoint switch with a value that is the UNC of a file. We decide what the file is called - its created on the first run of the script - but it has to have a "lpc" file type. In the screen shots below you can see that I am first of all collecting the top 100 entries in the system, security and application event logs on a PC. In the screen captures below  you can see the first run scans almost 141,000 rows of data in about 8 seconds, importing 100 rows into a csv file. This is pretty impressive but we don't really want to repeat that process every time we want more data from the event logs. You can see that in order to achieve this I specify a checkpoint file called AllLogs.lpc in the root of my C:\ drive.

LogParser "Select top 100 * into C:\AllLogs.csv from \\zoo-pc\system,\\zoo-pc\security,\\zoo-pc\application where eventtype = 4 order by timegenereated desc" -o:csv -icheckpoint:C:\AllLogs.lpc

Running the same query a second time, the checkpoint file is again used and there are 0 rows processed and 0 rows output in 0.08s. This means that the LogParser didn't go through all 141,000 rows again, it simply looked for new rows since the last run. There have been no new rows added to any of the event logs on my PC so, obviously, none were moved to my csv file. In the background the checkpoint file has been updated.

The third image, captured some time later, shows the results when I run the same LogParser query again. We see that 22 rows are processed and 10 are output, this means that the 3 event logs have been added to, with 22 events in total, and that 10 of the new rows qualify for my query and have been added to the AllLogs.csv file. Again, this only took 0.08s, a huge saving over the 8s in the first run. The checkpoint file will, again, have been updated.

If we now turn our attention to the data we are collecting in the C:\AllLogs.csv file we can make a very quick analysis of the data from LogParser by using the CHART output option. Simply write a LogParser query that returns some aggregated data and choose where you want the image to be created.

LogParser "SELECT SourceName, count(*) as [Number of Events] INTO C:\Events.gif FROM C:\AllLogs.csv Group By Sourcename order by [Number of Events] Desc" -o:CHART -charttype:Column3d

The resulting file - C:\Events.gif

Other chart types such as Pie and 3D column charts are available and you can create either gif or jpg file types - the LogParser help has all the details and links you need.

Well, that's it for part 4 of this series and we are almost done. In the 5th, and final, part I am going to visit ways of automating LogParser and using it to run in the background to assist with routine data processing. Be sure to be wearing your PowerShell hat next time!