cancel
Showing results for 
Search instead for 
Did you mean: 
Jfrazier
Level 18

Re: Delayed Alerts

images still coming up blank.

Can you save them locally and then upload them via the insert image tool ?

0 Kudos
j_a_catlin
Level 11

Re: Delayed Alerts

As a quick item to check, I would suggest doing a query against the alertlog table in the DB and seeing if the time of the actions and the alert triggering actually lines up with the time that node went down or the email was sent.  This could at least confirm that the alerts are triggering late and eliminate the possibility that it is just emails getting delayed somehow.

0 Kudos
jeremymayfield
Level 15

Re: Delayed Alerts

Have you verified there is no Delay in your SMTP server settings where in your email server is processing your requests from the Orion server?   seems like every update I get some that route to my outlook Spam folder. 

Also i would ask are you using the alerting feature in Orion or the Advanced alerting features on the Orion Server it self?  Advanced alerting has many features which can delay sending if misconfigured.

0 Kudos
jeremymayfield
Level 15

Re: Delayed Alerts

Have you Checked the Alert properties,

Name of alert:  

Email me when a Node goes down  

Description of alert:  

Paging

Type of Property to monitor

Node

Enabled(On/Off):  

ON  

Evaluation Frequency of alert:  

Every 15 seconds  

Severity of alert:  

Critical  

Alert Custom Properties: (0)

No Alert Custom Properties defined 

Alert Limitation Category  

No Limitation

Also make sure there are no time of Day settings against the alerts that would prevent alerts from being sent.

0 Kudos
cahunt
Level 17

Re: Delayed Alerts

Only if the DB is present on this box, but it is a small install... I think the CPU and RAM can take care of this small setup.

We are currently monitoring total of 91 nodes and 498 sensors.
0 Kudos
Jfrazier
Level 18

Re: Delayed Alerts

I wonder what the polling completion rate looks like.  I suspect it falls off through the day based on fazl-e-azeem​ comment that alerts are timely after a reboot and are delayed later.

I agree the cpu and ram are likely the big culprits. If they could increase it to at least 2 cores and 6GB of ram there should be improvement.

cahunt
Level 17

Re: Delayed Alerts

A few things off the top

                                   - Not sure about your changing IP Addresses on the node and what that's all about... maybe your setting them up with in band IP, and changing to an out of band address on the node, or another back door, loopback, etc.

                              

                                   - Your VM reset tells me you are losing resources, or as things start to run and query you have long run times on your filters, alters, reports, etc.... specifically alerts if too in dept or nested will cause you an issue of not being able to get through the entire DB before your 'Check this every X minutes (or Seconds)' is set to - Check your Server Application Logs for Issues there - and you can also check out link to an alert for long running queues... don't remember where i came across that, but you will want to edit the conditions - it's currently set with a check of 15 minutes.. so any query running longer than 15 minutes will trigger the email.... I am sure you want to edit that.

Long Running SQL Queue . AlertDefinition

                                  - Verify your actions and what time they trigger - you can use a simple query   - customize or adjust to make it fit or show off a specific alert, or other items..

SELECT TOP 1000 *

  FROM [AlertLog]

  where ActionType = 'EMail'

  order by LogDateTime desc

                                  -

0 Kudos
Highlighted
cahunt
Level 17

Re: Delayed Alerts

yes indeed... that reminds me to suggest checking polling...... via settings or with something like......

select

Engines.ServerName,Engines.IP, Engines.ServerType,

convert(varchar, round(nodes.systemuptime/60/60, 2, 1))+' hrs' as Uptime,

Engines.Elements as Elmts, Engines.Nodes, Engines.Interfaces as Int, Engines.Volumes as Vol,

c.custpolls as UnDP, a.samct as SAM,

N.Down_node, I.Down_Int, V.Down_vol, A2.Down_sam,

s.failed as noSNMP,

Engines.PollingCompletion as "%complt",

nodes.nodeid, nodes.CPUload as "%CPU", nodes.percentmemoryused as "%RAM",

e1.PropertyValue as NPM_Rate, e2.PropertyValue as SAM_Rate

from Engines

join nodes on engines.ip = nodes.ip_address

left join (select engineproperties.engineid, EngineProperties.PropertyValue from EngineProperties where engineproperties.propertyname = 'Orion.Standard.Polling') e1

  on engines.engineid = e1.engineid

left join (select engineproperties.engineid, EngineProperties.PropertyValue from EngineProperties where engineproperties.propertyname = 'APM.Components.Polling') e2

  on engines.engineid = e2.engineid

or possibly with - UDT Job Status report by polling engine

0 Kudos
rschroeder
Level 21

Re: Delayed Alerts

From your original comments, and from analysis by folks like Jfrazier​, it seems apparent the Orion VM does not have sufficient resources allocated to accomplish its tasks.

Assign it more memory and CPU and see if the problem decreases.

0 Kudos
fazl-e-azeem
Level 11

Re: Delayed Alerts

Events:

image001.png

0 Kudos