This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Advanced AppInsight-Style Log display in a SAM Monitor using Powershell

I hope people enjoy this one, as I have been meaning to post it for ages.  I have a few different version of the below log parsing code, once I built this first example.  While the code is advanced, it is relatively simple to take it and alter for your own purposes.

What we want to do:

  • return a count of errors since last poll  (so errors will not be double counted)
  • in the message:
    • display the error lines in order.
    • also display error log entries that occurred in a certain time window - giving a certain amount of history, and preventing errors from being a 'blink and you miss it' occurrence.

Example of log returned on the component screen

pastedImage_5.png

BONUS Points - Now Combine Logs From Various Servers into a Single View

pastedImage_4.png

Step 1 - Powershell

Take this example script and edit it to fit your situation.  As it stands, this script actually will parse a large number of log files - as long as they fit the form <base directory>\*\Logs\error.log (note the wild card in the path) The * part is used to return the log files as seperate output values, like this:

The code is designed to write a file with the last line read into a file with the same name as the log file - and suffixed with "sw_monitor_last_line.txt" so when testing this, you will probably want to delete this file to reset where the script is up to.

It takes two parameters - base log file path and time in minutes to always display log entries from.

# 2016-08-04  Mathew Frank  Initial Version
# 2016-09-08  Mathew Frank  New log entry detection. Old leg entries in window suffixed with ***
#
# #
# Logs will be found with the following path:   <base directory>\*\Logs\error.log
# Logs will be filtered to show only log entries no more than <minutes> old.
#


$WindowMinutes = $args[1]
$BasePath = $args[0]


$CutoffDate = (Get-Date).AddMinutes( -$WindowMinutes )


$out = @{}
$outCount = @{}


$dirs = dir -Directory $BasePath | select-object FullName, Name


#loop through every service
foreach ( $dir in $dirs ) {

#multi services / files - prefix with a number
if ($dir.Name -match ".+[0-9]+$") {
$svcNum = [regex]::match($dir.Name ,'([0-9]+)$').Groups[0].Value + ": "
$name =  $dir.Name -replace "[0-9]+$","" 
}else{
$name = $dir.Name
$svcNum = ""
}

$FullLogPath = $dir.FullName + "\logs\error.log"
$LastLineFile = $dir.FullName + "\logs\error.log.sw_monitor_last_line.txt"


if (Test-Path $FullLogPath  ) {
$LastLines = get-content $LastLineFile #get lines from last file check
$data = get-content $FullLogPath
$Lines = $data.Count
$Lines | Out-File $LastLineFile


$dataNew = $data | where {$_.readcount -gt $LastLines } #new log entries
$data = $data | where-object {$_.split("[")[0] -as [datetime] -gt $CutoffDate} #log entries in time window (may not be new this poll)


$NewLines = $Lines - $LastLines #number of new lines in log 
if ($outCount.ContainsKey( $name ) ) {
  $outCount[$name] += $NewLines
}else{
  $outCount[$name] = $NewLines
$out[$name] = ""
  }        


$x = 0 
$endOldLine = $data.count - $NewLines
$data | foreach {
if( $x -gt $endOldLine  ){
  $out[$name] += $svcNum + $_ + "<br />"
} else {
  $out[$name] += $svcNum + $_ + "***<br />"
}
$x++
}
}    
}
foreach( $v in $out.GetEnumerator() ) {
$name = "Message." + ($v.Name -replace " ","_")
$out = "<div style=""overflow:auto;height:80px;width:400px"">" + $v.Value + "</div>"
Write-Host $name ":" $out


$name = "Statistic." + ($v.Name -replace " ","_")
Write-Host $name ":" $outCount[$v.Name]
}

Some important highlights from above:

$FullLogPath = $dir.FullName + "\logs\error.log"
$LastLineFile = $dir.FullName + "\logs\error.log.sw_monitor_last_line.txt"

The above lines determind the log file name and the last line file(s).

$data = $data | where-object {$_.split("[")[0] -as [datetime] -gt $CutoffDate} #log entries in time window (may not be new this poll)

The above line extracts the data.  This log file had the date/time at the beginning of the line, then a space, followed by the log itself.  We need to work out  the date, so we can return X minutes worth of log entries

if( $x -gt $endOldLine  ){
  $out[$name] += $svcNum + $_ + "<br />"
} else {
  $out[$name] += $svcNum + $_ + "***<br />"
}

The above code is putting in the line breaks, and marking previously returned log data with ***

$out = "<div style=""overflow:auto;height:80px;width:400px"">" + $v.Value + "</div>"

This code just puts the log in a div with scrollbars - so as to avoid taking over the screen on the component pages.

Step 2 Bonus - Display these error logs and others

1. Create a custom query wherever you want to display the logs (like a NOC view)

2. SQL data source similiar to this:

select 
cast( cast(case when left(date_text,2)='12' and month(getdate()) = 1 then year(getdate())-1 else year(getdate()) end as varchar(4)) + '/' + date_text  AS datetime) Log_Time
,Node, Monitor
,case when out like '%***' then '<span style="color:grey;">' + out + '</span>' else out end
Log_Message
from (
select Node, Name as Monitor
,ltrim(substring(msg,24, len(msg)-23 )) out
,rtrim(ltrim(substring(msg,5,19))) date_text
from (
SELECT Node, Name, Label, msg
FROM
(
SELECT Node, Name, Label,
/*escape chars for valid xml, split on <br /> - which is '&lt;br /&gt; because just been escaped*/
cast('<X>'+REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(replace(substring(f.msg,52,len(f.msg)-57),'&','&amp;'),'<','&lt;'),'>','&gt;'),'"','&quot;'),'''','&#39;'),'&lt;br /&gt;','</X><X>')+'</X>' as XML) as xmlfilter from (
SELECT c.Name, dcs.Label, ISNULL(pe.ErrorMessage, deM.StringData) msg, ccs.ComponentID as ComponentID, n.caption Node
FROM dbo.APM_CurrentComponentStatus ccs WITH(NOLOCK)
left JOIN dbo.APM_PortEvidence pe WITH(NOLOCK) on ccs.ComponentStatusID = pe.ComponentStatusID
JOIN dbo.APM_ComponentExt c on (ccs.ComponentID = c.ID and c.ComponentType in (4, 5, 15, 7, 6, 14, 13, 21, 50, 11, 16, 18, 12, 29, 30, 31, 51, 10, 17, 2, 19, 20, 22, 23, 24, 25, 26, 27, 28, 32, 33, 34, 35, 36, 37, 38, 39, 42, 40, 41, 45, 46, 48, 44, 49, 58))
JOIN dbo.APM_Application a on ccs.ApplicationID = a.ID
JOIN dbo.Nodes n WITH(NOLOCK) on a.NodeID = n.NodeID
left join dbo.APM_DynamicEvidenceColumnSchema dcs on dcs.ComponentID = ccs.ComponentID and dcs.Type = 1
left join dbo.APM_DynamicEvidence de WITH(NOLOCK) on de.ComponentStatusID = ccs.ComponentStatusID and de.ColumnSchemaID = dcs.ID
left join dbo.APM_DynamicEvidence deM WITH(NOLOCK) on deM.ComponentStatusID = ccs.ComponentStatusID and deM.ColumnName = dcs.Name and deM.ColumnType = 0
WHERE c.Disabled = 0
and c.name in ('Worksite IM Email Svc Log Errors','Worksite FMA Log Errors')
) F
)F1
CROSS APPLY (
SELECT fdata.D.value('.','varchar(max)') as msg
FROM f1.xmlfilter.nodes('X') as fdata(D)
) O
) p3
where len(msg) > 0
) p4
order by 1

Now the above code looks a bit funky, but the only part to worry about if you stay with error lines split using "<br />"  is this line:

and c.name in ('Worksite IM Email Svc Log Errors','Worksite FMA Log Errors')

This is the name of the components you want to return.  This code will find ALL instances of these components, and return them as a single dataset.

What is it actually doing?  it is returning a single line for each error log SAM component, then splitting those into seperate rows, THEN finally sorting the result.  There is a minor problem with this code - when processing massive log data, it will choke the SQL Server - and time out.  When I get time I will alter the query such that it will limit the number of lines it will process to avoid this (in my experience this has happened maybe twice in 3 years though)

3.Setup the grouping and fields.  In my example I format log_time as DateTime - and group by this.  Also Log_Message is HTML formatted.

pastedImage_1.png