Loss of connection to database, reconnecting (feature requests?)

Question

We had a power issue in our datacenter this weekend and the database server was unavailable.  We noticed this as Orion's front-end was erroring out with an IIS-type error page.  The backend would prompt that it couldn't reach the database or didn't have permissions, etc.  When the database server was restored, Orion's website became functional again and everything seemed fine.  We didn't notice until the morning that nothing was being written to the database.  We had to reboot the server to restore functionality.  I'm sure stopping/starting the Orion services may have accomplished the same task.

I spoke with our database administrator and he said it didn't appear as if the Orion application was checking for connectivity to the database.  Once it couldn't reach it, when it could, it wasn't coded to re-authenticate and establish communication.  If the database server takes a dump again, it looks like we need to manually re-connect Orion to it and I was wondering if this was by design, or if a feature request was appropriate to have Orion check for connectivity at an interval, or each time a WRITE was performed.

IF no connection during a write, kick off a connection subroutine, etc.

ecornwell · Answer

We've seen the same thing.  There was a problem with our SQL server where the main process would hang and our DBA's ended up killing it.  Our server admin noticed an error on our Orion server and when it went away everything was happy.  I checked the the polling engine stats and it had been over 17 hours since it talked to the database but everything seemed fine.

I ended up killing then restarting the NetPerfMon service and it came back.  (Note: I always have to do an end process on it, it NEVER shuts down on its own.)

bcole · Answer

I noticed the same problem but really the horrible part is all of the alerts go off so I get several thousand messages all at once.  My BB can't keep up.

gbrance · Answer

I have seen this same issue.  We have our Orion Database on a SQL cluster.  Sometimes during a cluster failover Orion would work but it wouldn't record any data.  I have adjusted the database timeout to help prevent this during a cluster failover.