9 Replies Latest reply on Aug 14, 2014 10:27 AM by curtisi

    LEM Database not running

    diimack

      Hi All,

       

      I cannot get to start/restart the database. What could possibly be the cause?

       

      Regards,

        • Re: LEM Database not running
          curtisi

          If you open the CMC shell, go to Manager and then do a WATCHLOG, what does the manager log say?

           

          Is the disk full?  (Go to APPLIANCE --> DISKUSAGE)

           

          Is the store for the VM disk full?

           

          You should probably be opening a ticket with Support.  All of the support numbers are listed on our website at http://www.solarwinds.com/company/contact.aspx

           

          Once you reach the menu, pick option 3 for Technical Support.  Have your Product and SolarWinds ID (SWID) ready

          1 of 2 people found this helpful
            • Re: LEM Database not running
              diimack

              Thanks Curtisi & Tmiller.

               

              For Disk usage, I have the following result:

               

              Disk Usage:

              TriGeo: 22% (611M/3.0G)

              OS: 45% (1.3G/3.0G)

              Logs/Data: 44% (96G/234G)

              Temp: 22% (1.2G/5.9G)

              Database Queue(s): 1.1G (1509340 alerts queued, 119802 alerts waiting in memory)

              Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

              Console Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

              EPIC Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

              Logs: 211M

              Tool Profiles Message Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

               

              For the WATCHLOG, I have the following:

               

                      at com.trigeo.manager.license.LemLicenseManager.getMaximumNodeLimit(LemLicenseManager.java:1343)

                      at com.trigeo.manager.license.LemLicenseManager.availableNodesRemaining(LemLicenseManager.java:1275)

                      at com.trigeo.manager.license.LemLicenseManager.isLicenseMaxedOut(LemLicenseManager.java:1609)

                      at com.trigeo.manager.treelayer.HierarchyTree.isLicenseMaxedOut(HierarchyTree.java:353)

                      at com.trigeo.manager.communications.manager.ComNetworkInstall.hasAvailableSpace(ComNetworkInstall.java:415)

                      at com.trigeo.manager.communications.manager.ComNetworkInstallHandler.handleMessage(ComNetworkInstallHandler.java:75)

                      at com.trigeo.manager.communications.manager.ComNetworkInstall$CCWorker.run(ComNetworkInstall.java:700)

                      at java.lang.Thread.run(Thread.java:619)

                      at com.trigeo.util.TriGeoThread.run(TriGeoThread.java:57)

              ;

              (Tue Aug 12 13:07:12 PGT 2014) EE:ERR [com.trigeo.core.database.repository.VerticaSQL v23476] {BBS:DequeueToDB-1:47} Disconnect. Could not get connection to default database  [DEFAULT DB CODE=08004]

              com.vertica.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. EXCEPTION: com.vertica.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

              ...

                   

              Is it a license issue, or a connection issue, or something else? I just do not understand.

               

              Thanks

                • Re: LEM Database not running
                  curtisi

                  Okay, so not a disk full.  It appears that the database isn't running on the LEM, so events aren't getting archived.  Your temp space is being used to store events while the manager collects events with no place to put them. Eventually, that's going to be full and the LEM will start rotating that space, which will lead to gaps in the logs.

                   

                  Now, it looks like you have an older LEM version (5.5 or prior), but I'm guessing from the size of the partitions this is a virtual appliance.  Since it's unlikely that you did an L4 deployment (separate database and manager appliances) in a virtual environment, it sounds like the database on the LEM has stopped for some reason.  You really should open a support ticket so we can try to repair the database and get it running before you start dropping data.

                   

                  Once it's fixed, you should move to 5.6 and then 6.0, which will mean migrating to the new database standard for the LEM.

                  1 of 2 people found this helpful
                    • Re: LEM Database not running
                      diimack

                      Curtisi,

                       

                      Thanks a lot for the diagnosis and the advice - It's very helpful to a beginer like me..:)

                      And yes, you are correct in everything you said: We have LEM v5.5 running on a Virtual Appliance, and not a L4 deployment.

                       

                      Regards,

                • Re: LEM Database not running
                  tmiller_hockey

                  Disk full would be my guess as well.  It doesn't like that at all.

                  • Re: LEM Database not running
                    diimack

                      Hi All,

                     

                    After activating the license, we have the following information displayed on the license section of the LEM console. Isn't it incorrect to have 0 license installed? Could that be the cause for the DB not running? Will upgrading/resinstall resolve it?

                    AA.png
                       

                    Thanks..

                      • Re: LEM Database not running
                        curtisi

                        The LEM0 is a known issue with using a LEM 5.6+ license on a LEM 5.5 or older system.  The upgrade would sort that out, but you still want support to sort out the DB before you start an upgrade.  Or you can decide to nuke the DB and start from scratch, which eliminates the whole problem.  That will be a choice in the 5.6 upgrade.

                          • Re: LEM Database not running
                            diimack

                            Thanks Curtisi..

                            So what you are saying is: we can iether have support fix the database before we upgrade (so that we don't lose any data) or we can just upgrade without having the DB fixed, and in the upgrade process, have the DB nuked and start from scratch..

                             

                            We are already in contact with support that's why I'm seeking advice for the way foward.

                             

                            Cheers!