LEM Database not running

If you open the CMC shell, go to Manager and then do a WATCHLOG, what does the manager log say?

Is the disk full?  (Go to APPLIANCE --> DISKUSAGE)

Is the store for the VM disk full?

You should probably be opening a ticket with Support.  All of the support numbers are listed on our website at http://www.solarwinds.com/company/contact.aspx

Once you reach the menu, pick option 3 for Technical Support.  Have your Product and SolarWinds ID (SWID) ready

Disk full would be my guess as well.  It doesn't like that at all.

Thanks Curtisi & Tmiller.

For Disk usage, I have the following result:

Disk Usage:

TriGeo: 22% (611M/3.0G)

OS: 45% (1.3G/3.0G)

Logs/Data: 44% (96G/234G)

Temp: 22% (1.2G/5.9G)

Database Queue(s): 1.1G (1509340 alerts queued, 119802 alerts waiting in memory)

Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

Console Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

EPIC Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

Logs: 211M

Tool Profiles Message Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

For the WATCHLOG, I have the following:

        at com.trigeo.manager.license.LemLicenseManager.getMaximumNodeLimit(LemLicenseManager.java:1343)

        at com.trigeo.manager.license.LemLicenseManager.availableNodesRemaining(LemLicenseManager.java:1275)

        at com.trigeo.manager.license.LemLicenseManager.isLicenseMaxedOut(LemLicenseManager.java:1609)

        at com.trigeo.manager.treelayer.HierarchyTree.isLicenseMaxedOut(HierarchyTree.java:353)

        at com.trigeo.manager.communications.manager.ComNetworkInstall.hasAvailableSpace(ComNetworkInstall.java:415)

        at com.trigeo.manager.communications.manager.ComNetworkInstallHandler.handleMessage(ComNetworkInstallHandler.java:75)

        at com.trigeo.manager.communications.manager.ComNetworkInstall$CCWorker.run(ComNetworkInstall.java:700)

        at java.lang.Thread.run(Thread.java:619)

        at com.trigeo.util.TriGeoThread.run(TriGeoThread.java:57)

;

(Tue Aug 12 13:07:12 PGT 2014) EE:ERR [com.trigeo.core.database.repository.VerticaSQL v23476] {BBS:DequeueToDB-1:47} Disconnect. Could not get connection to default database  [DEFAULT DB CODE=08004]

com.vertica.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. EXCEPTION: com.vertica.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

...

Is it a license issue, or a connection issue, or something else? I just do not understand.

Thanks

Okay, so not a disk full.  It appears that the database isn't running on the LEM, so events aren't getting archived.  Your temp space is being used to store events while the manager collects events with no place to put them. Eventually, that's going to be full and the LEM will start rotating that space, which will lead to gaps in the logs.

Now, it looks like you have an older LEM version (5.5 or prior), but I'm guessing from the size of the partitions this is a virtual appliance.  Since it's unlikely that you did an L4 deployment (separate database and manager appliances) in a virtual environment, it sounds like the database on the LEM has stopped for some reason.  You really should open a support ticket so we can try to repair the database and get it running before you start dropping data.

Once it's fixed, you should move to 5.6 and then 6.0, which will mean migrating to the new database standard for the LEM.

Curtisi,

Thanks a lot for the diagnosis and the advice - It's very helpful to a beginer like me..:)

And yes, you are correct in everything you said: We have LEM v5.5 running on a Virtual Appliance, and not a L4 deployment.

Regards,

  Hi All,

After activating the license, we have the following information displayed on the license section of the LEM console. Isn't it incorrect to have 0 license installed? Could that be the cause for the DB not running? Will upgrading/resinstall resolve it?


   

Thanks..

The LEM0 is a known issue with using a LEM 5.6+ license on a LEM 5.5 or older system.  The upgrade would sort that out, but you still want support to sort out the DB before you start an upgrade.  Or you can decide to nuke the DB and start from scratch, which eliminates the whole problem.  That will be a choice in the 5.6 upgrade.

Thanks Curtisi..

So what you are saying is: we can iether have support fix the database before we upgrade (so that we don't lose any data) or we can just upgrade without having the DB fixed, and in the upgrade process, have the DB nuked and start from scratch..

We are already in contact with support that's why I'm seeking advice for the way foward.

Cheers!

Yeah, that's the options.  Starting the upgrade with a broken database might work, but I wouldn't be willing to promise you anything at that stage.