VMware Event Logs and PowerCLI

If you are responsible for VMware servers, one of your management tasks should probably be to keep an eye for problems in the event log. In the Windows world we have a few PowerShell cmdlets at our disposal for remote event log management. Obviously those won’t work for a VMware ESXi server. What we do have though, is PowerCLI, which we’ve been covering lately on the site. Let me show you how to retrieve event log information from your VMware servers.

Getting Event Log Types

The first thing you need to do is identify the type of event log. I already have PowerCLI loaded and am connected to my default ESXi server. Using Get-LogType will display the available logs.

​ PS C:\> get-logtype
Key                  Summary
---                  -------
hostd                Server log in 'plain' format
vmkernel             Server log in 'plain' format
vpxa                 vCenter agent log in 'plain' format

The logs are in plain text as you can see in the summary, which is good because as I’ll demonstrate in bit you might have to stretch your string parsing skills. From this list you need to make note of the Key. You will need this to get the actual entries for the corresponding log.

Getting Event Log Entries with PowerCLI

PowerCLI has a Get-Log cmdlet to do the heavy lifting of retrieving the contents of the event log. At a minimum all you need to do is specify the log key, which you get with Get-LogType. I’m going to get entries from hostd.

​ PS C:\> $hostd = get-log hostd
PS C:\> $hostd
Entries
-------
{2013-11-07T16:45:02.013Z [FFE07D20 verbose 'SoapAdapter'] Responded to ...

Be careful, as the key name, e.g. hostd, is case sensitive. You need to specify the key exactly as you see it with Get-LogType.
So I have an object, $hostd, that has a single property, Entries, which is a collection of strings. These are the lines of text from the event log.

​PS C:\> $hostd.entries.count
12741

That’s a lot of lines. In order to read the log, I have to get the items from Entries as shown below in Figure 1.
VMware Event Logs and PowerCLI
The better approach, in my opinion, is to expand the entries as they are being collected.

​ PS C:\> $hostd = get-log hostd | Select -ExpandProperty Entries
PS C:\> $hostd.count
12781

Now $hostd is a copy of the event log. I could pipe it to Out-File to save a local copy. Or further parse it.
It is also possible to get a subset of log entries. You can specify the number of lines to return.

​ PS C:\> (get-log hostd -NumLines 1 ).Entries
2013-11-07T16:45:02.013Z [FFE07D20 verbose 'SoapAdapter'] Responded to service state request

You can also specify at what line number. The default is 1. The log starts with the oldest entry.

​ PS C:\> get-log hostd -NumLines 3 -StartLineNum 1000

Or if you don’t specify the number of lines, the cmdlet will get all event log entries starting at the given line number.

​ PS C:\> get-log hostd -StartLineNum 12770 | Select -expand entries

You can see my result in Figure 2.
VMware Event Logs and PowerCLI
The catch here is that you don’t know how many lines you have unless you retrieve the entire log. So you might as well grab everything as I did earlier.

 

Parsing the Log File

Another important thing you should know is that not every item in your log (this is $hostd in my example) is an event log entry. Some entries have multiple lines.

​ 2013-11-07T16:47:29.281Z [6829DB90 info 'Vmomi' opID=FE942AB6-00001029] Result:
--> (vmodl.fault.RequestCanceled) {
-->    dynamicType = <unset>,
-->    faultCause = (vmodl.MethodFault) null,
-->    msg = "",
--> }

This is a single entry but would count as six lines. Even so, with everything as an array you could get the last part of the file.

​ PS C:\> $hostd[-1..-20]

This should be the 20 or so last lines of the log file which will be the most recent entries.
VMware Event Logs and PowerCLI log file
 
Or use Select-String to find relevant information.

​ PS C:\> $hostd | select-string "VMsvc" | select -last 5

Figure 4 below shows the five most recent entries that have VMSvc in the entry.
VMware Event Logs and PowerCLI vmsvc
The last thing I want to show you is a way to convert that funky time stamp you see at the beginning of each entry into a more user-friendly and local time. With a little regular expression magic, you can replace that string with a friendlier date format.
To begin, you need a regular expression pattern for the date/time format.

​ [regex]$rx="^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z"

Then you can pipe the entries to ForEach-Object.

​ $hostd[0..10] | foreach {
if ($rx.Ismatch($_)) {
[datetime]$dt = $rx.match($_).Value
$rx.Replace($_,$dt.tostring())
}
else {
$_
}
} #foreach

If the line of text matches the regex pattern, then the matching value is saved as a datetime object, $dt, and then replaced in the line. The replace line gets written to the pipeline. If there is no match, then the line is simply passed back through. You can see the result for a sample of lines in Figure 5.
VMware Event Logs and PowerCLI vmsvc
Putting it all together you might want to run a PowerCLI expression like this:

​ $hostd = get-log hostd | Select -ExpandProperty Entries |
foreach -begin {
[regex]$rx="^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z" } -process {
if ($rx.Ismatch($_)) {
[datetime]$dt = $rx.match($_).Value
$rx.Replace($_,$dt.tostring())
}
else {
$_
}
} #foreach

Now $hostd has the entire log and the time stamps are local. I can parse or filter as I need, or save the results to a local file.

​ PS C:\> $hostd | out-file c:\work\hostd.log -Encoding ascii

 
Because VMware servers obviously are not running Windows, it takes a bit more effort to retrieve and analyze logs. But it is still very doable with PowerShell and PowerCLI. You simply might have to be a bit clever, such as using my regex trick to “convert” the time stamp, to get the most out of the cmdlet.