2 Replies Latest reply on Apr 22, 2016 5:07 PM by patches_mcmatt

    LEM: Temp filesystem is over 90% full

    patches_mcmatt

      I'm new at managing our company's Log & Event Manager application and am trying to discover the cause of a problem that I've noticed over the past couple of days where LEM will display an incident notification stating "managermonitor warning! disk usage: the temp filesystem is over 90% full". The incident can be viewed under the Security > Incidents filter.

       

      I've figured out how to clear the temp directory and was able to do so successfully yesterday. Upon arriving to work this morning, I noticed that the temp directory is full again, but I don't know why.

       

      Here is the output of the diskusage command.

       

      cmc> appliance

      cmc::acm# diskusage

      Checking Disk Usage (this could take a moment)... ....oo.oo.oo.oo.oo.oo.oo.

      Partition Disk Usage:

              LEM:             43% (1.2G/3.0G)

              OS:              46% (1.3G/3.0G)

              Logs/Data:       90% (199G/234G)

              Temp:            95% (5.3G/5.9G)

      Database Queue(s): 5.1G (12679286 alerts queued, 187196 alerts waiting in memory)

      Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

      Console Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

      DataCenter Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

      EPIC Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

      Forensic Database Queue: 2.1M (0 data queued, 0 data items waiting in memory)

      Logs: 11G

      Tool Profiles Message Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)

       

      When I use the cleantemp command and look through the directories in /tmp, I see that only one of the directories holds nearly all of the data that is filling up the temp space. That directory is called "Standard_Local_Database". It now contains 641 ".qa" files after having been cleared out around 24 hours ago. Based on the timestamps, it appears that a new file is created and stored here once per minute.

       

      What is the best course of action for troubleshooting what's causing the temp directory to fill up so quickly? Thanks.