3 Replies Latest reply on Apr 7, 2014 11:04 AM by nazar_hryt

    AlertCentral Not Processing Orion-SAM Alerts

    david.noll@elementis.com

      For some reason, SAM and AlertCentral are not playing together nicely.  When I look at the mail file for my SWAlerts account, it seems to be recycling the same set of alerts over and over again.  Its almost as if it became overloaded and the data became corrupted.  When I look at SAM Subsriptions table, it says failed delivery attempts of 25.   There are some ~700 messages in the email file and it's the same thing over and over.

       

      "Unable to acknowledge this alert (alert was CLOSED by admin at XX:XX PM) "

       

       

      I am attaching the logs.  The errors that stands out to me is this section is ALERT_NOT_FOUND - Alert with ID 0 and Persistent Store is Full:

       

      09:00:17.431 [http-bio-8080-exec-8] ERROR c.s.o.o.l.i.OrionToAlertCentralSyncHandler - addNoteToAlert failed: ALERT_NOT_FOUND - Alert with ID=0 not found.

      09:00:19.907 [ActiveMQ Task-8]  INFO o.a.activemq.broker.region.Queue - Usage(default:store:Producer:store:queue://com.solarwinds.oncall.notification:store) percentUsage=0%, usage=0, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:store:Producer:store) percentUsage=0%, usage=0, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%, usage=536892973, limit=536870912, percentUsageMinDelta=1%: Persistent store is Full, 100% of 536870912. Stopping producer (notifQueueConnection-49800-1396270305005-7:2:623:1) to prevent flooding queue://com.solarwinds.oncall.notification. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 164s)

      09:00:47.488 [http-bio-8080-exec-20] ERROR c.s.o.o.l.i.OrionToAlertCentralSyncHandler - addNoteToAlert failed: ALERT_NOT_FOUND - Alert with ID=0 not found.

      09:00:48.058 [ActiveMQ Task-3]  INFO o.a.activemq.broker.region.Queue - Usage(default:store:Producer:store:queue://com.solarwinds.oncall.email:store) percentUsage=0%, usage=0, limit=67108864, percentUsageMinDelta=1%;Parent:Usage(default:store:Producer:store) percentUsage=0%, usage=0, limit=67108864, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=103%, usage=69615377, limit=67108864, percentUsageMinDelta=1%: Persistent store is Full, 100% of 67108864. Stopping producer (emailQueueConnection-49800-1396270305005-3:1:1:1) to prevent flooding queue://com.solarwinds.oncall.email. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 517s)

      09:00:48.559 [ActiveMQ Task-7]  INFO o.a.activemq.broker.region.Queue - Usage(default:store:Producer:store:queue://com.solarwinds.oncall.workflow:store) percentUsage=0%, usage=0, limit=201326592, percentUsageMinDelta=1%;Parent:Usage(default:store:Producer:store) percentUsage=0%, usage=0, limit=201326592, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=103%, usage=208922590, limit=201326592, percentUsageMinDelta=1%: Persistent store is Full, 100% of 201326592. Stopping producer (workflowQueueConnection-49800-1396270305005-5:2:1:1) to prevent flooding queue://com.solarwinds.oncall.workflow. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 517s)

      09:00:52.067 [ActiveMQ Task-8]  INFO o.a.activemq.broker.region.Queue - Usage(default:store:Producer:store:queue://com.solarwinds.oncall.notification:store) percentUsage=0%, usage=0, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:store:Producer:store) percentUsage=0%, usage=0, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%, usage=536895961, limit=536870912, percentUsageMinDelta=1%: Persistent store is Full, 100% of 536870912. Stopping producer (notifQueueConnection-49800-1396270305005-7:2:623:1) to prevent flooding queue://com.solarwinds.oncall.notification. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 196s)

      09:01:20.153 [ActiveMQ Task-3]  INFO o.a.activemq.broker.region.Queue - Usage(default:store:Producer:store:queue://com.solarwinds.oncall.email:store) percentUsage=0%, usage=0, limit=67108864, percentUsageMinDelta=1%;Parent:Usage(default:store:Producer:store) percentUsage=0%, usage=0, limit=67108864, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=103%, usage=69615377, limit=67108864, percentUsageMinDelta=1%: Persistent store is Full, 100% of 67108864. Stopping producer (emailQueueConnection-49800-1396270305005-3:1:1:1) to prevent flooding queue://com.solarwinds.oncall.email. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 549s)

      09:01:20.654 [ActiveMQ Task-7]  INFO o.a.activemq.broker.region.Queue - Usage(default:store:Producer:store:queue://com.solarwinds.oncall.workflow:store) percentUsage=0%, usage=0, limit=201326592, percentUsageMinDelta=1%;Parent:Usage(default:store:Producer:store) percentUsage=0%, usage=0, limit=201326592, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=103%, usage=208922590, limit=201326592, percentUsageMinDelta=1%: Persistent store is Full, 100% of 201326592. Stopping producer (workflowQueueConnection-49800-1396270305005-5:2:1:1) to prevent flooding queue://com.solarwinds.oncall.workflow. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 549s)

      09:01:24.160 [ActiveMQ Task-8]  INFO o.a.activemq.broker.region.Queue - Usage(default:store:Producer:store:queue://com.solarwinds.oncall.notification:store) percentUsage=0%, usage=0, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:store:Producer:store) percentUsage=0%, usage=0, limit=536870912, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%, usage=536899308, limit=536870912, percentUsageMinDelta=1%: Persistent store is Full, 100% of 536870912. Stopping producer (notifQueueConnection-49800-1396270305005-7:2:623:1) to prevent flooding queue://com.solarwinds.oncall.notification. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 228s)

      09:01:48.173 [http-bio-8080-exec-27] ERROR s.data.serialization.JsonSerializer - Error serializing query results

      java.lang.ArrayIndexOutOfBoundsException: null

      at java.lang.System.arraycopy(Native Method) ~[na:1.7.0_04]

      at org.codehaus.jackson.util.TextBuffer.buildResultArray(TextBuffer.java:699) ~[jackson-core-asl-1.9.6.jar:1.9.6]

      at org.codehaus.jackson.util.TextBuffer.contentsAsArray(TextBuffer.java:374) ~[jackson-core-asl-1.9.6.jar:1.9.6]

      at org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:150) ~[jackson-core-asl-1.9.6.jar:1.9.6]

       

      Is there a way to just reset the Alerts table and start over with this data?  I don't need the old alerts.