Alerting Concerns in Orion NPM 9.5

Please apply SP2 to fix this issue, see here

All comments

FormerMember

I have seen this problem on at least 2 of my sites. I have around 10-12 orion installs, and 2 of them have shown this problem over the last 12 months. the fact that it was so random meant that i did not log it out with solarwinds.

i have considered restarting the alerting service each night but not sure that is satisfactory and not sure how the engine works, so that might generate an alert for all DOWN nodes when the servie restarts which would result in duplicate alerts over time....

anyone else?

I currently have a case open for this: 100027. I've supplied Diag files and event viewer info. Seems to be a random halt in the Alerting engine or some failure accessing the Database stopping the alerts. I haven't heard from support since 7am so I hope they are busy working on a resolution.

ElisabethZakes

Yes, they're working on it. Your technician is in the Cork (Ireland) office, so you'll likely hear from him in his morning.

breener96

I also had issues with alerting after upgrading to 9.5. I had to reinstall the alerting service.

I ahd also been having difficulty with even basic alerts working correctly. I found that testing did not seem to function.

I looked at the alert log and discovered an intermittent database connection error. I performed a repair of the SW alerting service and that appeared to solve the problem. However, it seems to be back again after only a day.

Here is the current text from the log:

2009-06-16 10:01:02,750 [MainTaskThread] INFO All - Alert Engine Starting. Running Version 9.5.0.0.
2009-06-16 10:18:40,140 [MainTaskThread] ERROR All - Error in SetupDBConnection System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at AlertingEngine.SWAlertingEngine.SetupDBConnection(SqlConnection& DBConnection, SqlCommand& DBCommand, SqlDataReader& DBReader)
2009-06-16 10:18:40,203 [MainTaskThread] ERROR All - Exception in MainTask Loop - Unable to create DB connection
2009-06-16 10:18:48,171 [MainTaskThread] INFO All - Alert Engine Stopping

Yeah, something is messed up indeed. I show the alerting service is running, but a simple node down alert will not fire until I stop and start the service. Then, the alert immediately shows up in System Manager.

Since my Tech seem to only be working his timezone and answering requests while I am not in the office, this morning I did a full repair from the add/remove programs and re-installed the SP1. This seems to have helped with the communication errors in the logs, and the alerting.

Maybe the SW mod can have my ticket re-assigned to someone in the states so that I can work with someone with the same work hours?

r0berth1

I had the same problem untill i installed SP1 for 9.5.

Please apply SP1 and if still an issue, please open a support case

ElisabethZakes

Maybe the SW mod can have my ticket re-assigned to someone in the states so that I can work with someone with the same work hours?

cvinka, I'll send you an e-mail about this.

freemen

Brandon,

Re-installing and applying SP1 seems to have ficed it again, but the question is will it remained fixed:)

I tested again using my advanced node down alert. The alert eval frequency is set to 1 minute. I took down a node manually. I got two emails 2 minutes apart. Will I continue to get emails 1 or 2 minutes apart aas long as the node stays down? Id that the way the alert eval works?

FormerMember

I worked with technical support yesterday which included a "repair" of 9.5 and adding SP1. We experienced the same issue beginning again at 3:45 this morning. Kind of frustrating

FormerMember

Technical support indicated a code fix is in the works.

See http://www.solarwinds.com/NetPerfMon/SolarWinds/wwhelp/wwhimpl/js/html/wwhelp.htm#href=OrionAPMPHComponentTypesSqlQA.htm

Does anyone know if the issue has been identified and and ETA on the fix? Right now the software can't be trusted to monitor and that a very bad thing... I've been asked to back out this version if we don't see a fix in the near future, and I know that's not going to make me any happier.

dperdue

We have identified the problem and are working on a fix.

The problem happens when a connection to the database is lost temporarily. The work-around for now is to restart the alert service after database connectivity is restored.

r0berth1

I have a few alerting problems with 9.5 but they went away once i installed SP1. The difference may be that stat seeker is alerting instantly when the circuit goes down. NPM waits 1 min before it alerts by default. I have mine set to alert after the circuit has been in a down state for 3 min. It has been accurate so far. I was calling my locations as soon as i get the alerts and verified the times and they were correct. I also have the locations calling me as soon as the locations go down so i know I am getting all of the alerts. I have one more week to of operating like this before we go back to where the locations dont call anymore just to be sure orion is working properly since the upgrade. but so far so good.

freemen

dperdue,

This is perhaps a little off topic, but can we not test the SQL database connection using APM? If so, which monitor or application specifically would be used? Thanks.

ShamChauthani

The SQL User Experience Monitor is the best suited for this task.

You will want to use the monitor to connect to Orion DB and run a query (perhaps on the nodes table) to ensure connectivity to the DB.

Please apply SP2 to fix this issue, see here

Initial testing of the hotfix looks good. I stopped and then restarted the SQL instance on my server and then tested a simple node down alert. All functionality seems fine. Will test a few more times today.

breener96

I agree that I cannot trust Solarwinds 9.5 right now... everyday my alerting just stops working and I have to restart the alerting service. This is a HUGE problem and is unacceptable for an enterprise application. I had also had other issues, which were fixed, but not without a lot of time dedicated to fixing it.

I also do not want to revert back to 9.1 SP5, but I might have to soon if it is not fixed.

I am terribly disappointed with this release as I have had multiple issues since its upgrade. I care more about the reliability of the application than I do with the addition of new features. I know that I will never upgrade NPM during its early stages again.

Some advice to Solarwinds - do not rush your releases and be sure that it is reliable. Perform better testing... reliability is sooooo much more important than new features! If I wasnt dependednt on this application right now, then I would be reviewing alternative products.

dperdue

breener96,

Have you applied the hotfix referenced above? This should solve the alerting issue.

I loaded the hot fix as part of my case ticket update yesterday afternoon. I think I prefer doing the scheduled stop/start of the service. The fix seems to be keeping the service from stopping now.

One scare I had last night: I logged in to the web console to verify everything was ok and saw in the even summary a total for devices removed equaled that of the devices we have in the system! I'm not sure why or how this could be reported, just happy to say everything was still in the system and being polled.

I have noticed with the fix the the Business layer seems to be debugging more and I still see Syslog and Trap DB errors in the event log. I really hope SP2 comes out Monday and resolves all these issues. Management is really unhappy, and I'm spending way too much time on this.

ElisabethZakes

cvinka, I'm glad the hotfix is working for you on the service staying up!

I believe your support technician is working with you on the syslog/trap issue since it's not related to the Alert hotfix. And SP2 is still due to be out very shortly! Watch for that announcement on the forum soon.

Thanks to everyone who's been posting about the alerts and giving input so we could get this resolved quickly for you!

r0berth1

I know that I said above that i quit having problems once I installed SP1, but I have a correction or two to add to that statement. The alerting problems that I was having was that the alerts were not running before SP1. Now, they are working fine, but had my first major outtage last night and found that about 50% of the alerts were accurate, the rest either sent out alert when the circuit never went down, or the circuit went down but I didnt get an alert. So now i am unsure if I can trust Orion as much as I did per 9.5. Unfortunatly for me, I dont have any other software to rely on for the alerts. So SW needs to step up their testing efforts with SP2 or SP3.

r0berth1, did you apply the hotfix?