I'm new at managing our company's Log & Event Manager application and am trying to discover the cause of a problem that I've noticed over the past couple of days where LEM will display an incident notification stating "managermonitor warning! disk usage: the temp filesystem is over 90% full". The incident can be viewed under the Security > Incidents filter.
I've figured out how to clear the temp directory and was able to do so successfully yesterday. Upon arriving to work this morning, I noticed that the temp directory is full again, but I don't know why.
Here is the output of the diskusage command.
cmc> appliance
cmc::acm# diskusage
Checking Disk Usage (this could take a moment)... ....oo.oo.oo.oo.oo.oo.oo.
Partition Disk Usage:
LEM: 43% (1.2G/3.0G)
OS: 46% (1.3G/3.0G)
Logs/Data: 90% (199G/234G)
Temp: 95% (5.3G/5.9G)
Database Queue(s): 5.1G (12679286 alerts queued, 187196 alerts waiting in memory)
Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
Console Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
DataCenter Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
EPIC Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
Forensic Database Queue: 2.1M (0 data queued, 0 data items waiting in memory)
Logs: 11G
Tool Profiles Message Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
When I use the cleantemp command and look through the directories in /tmp, I see that only one of the directories holds nearly all of the data that is filling up the temp space. That directory is called "Standard_Local_Database". It now contains 641 ".qa" files after having been cleared out around 24 hours ago. Based on the timestamps, it appears that a new file is created and stored here once per minute.
What is the best course of action for troubleshooting what's causing the temp directory to fill up so quickly? Thanks.
