Alert that will do Dynamic Remediation for Drive Space

I am trying to work on some basic remediation tasks in Solarwinds and would like to start basic. If I get a drive space alert run script on server to clear certain directories, if this works clear the alert, if not fire off the alert. I saw something about this awhile back in a webinar or THWACK thing.

1.Alert checks if Drive space hits 95% utilized.

2.Node drive space his 95% utilized run PS Script to delete data out of certain directories.

3. PS Script runs, deletes what it can.

4. If this cleared up enough drive space do not send out an alert.

5. If drive space is still over 95% utilized send out an alert.

Can I do this in one alert or am I over simplifying this?

I am assuming we can use Execute an External Program.

I am going over my notes on this to see if I can figure it out. Hoping this is simple enough to start out with and once I get this figured out can expand out more.

Thanks,
Dave

Find more posts tagged with

Automation

drive space

Alerts

Accepted answers

All comments

jm_sysadmin

I have run tasks directly from Alerts for cleanup. Your idea works, you might need to pay attention to what rights the automation has when it executes.
We have moved to calling external jobs to get the work done. Clean up task can tend to add a lot of work compared to normal alerts so be careful about that. In our environment it didn't scale well. Specifically, our disk alerts came in groups when large updates rolled out. Today we don't call the cleanup script from the alert we call an API of our automation platform to do the cleanup.

Today we use a tool, Rundeck Runbook Automation (but there are lots of other tools that may fit your organization better). All the IS 'home grown' scripts built for reuse run from here, some scheduled, some ad hoc, some via API's triggered in other tools. This simplifies logging, allows for delegation down to staff that otherwise couldn't perform these approved tasks, integrated with Vaults for key/cert/passwords that handle rotation, and offloads the work from Solarwinds.

martian_monster

Thank you for your insights on this. What sounds not too complicated can usually get interesting with all the different variables out there. After looking at this for a bit I changed directions and started using the alerting engine to assign custom properties.

ihigareda

Hello @martian_monster i hope you´re well.

Were you able to do it through an external call?

ronan.tonilon

Disclaimer:
The content posted herein are provided as a suggestion or recommendation to you for your internal use. The information set forth herein may come from third party website or customers. SolarWinds is not liable for any downtime or any issue that may occur if you perform the following suggestions on the link provided. Your organization should internally review and assess to what extent, if any, such custom scripts or recommendations will be incorporated into your environment.
Please try this first in a lab environment, before implementing to your production.

You can open a support ticket to get insights, but please take note of our Working with Support Policy, customizations are not supported especially if this would involved custom scripts or queries, support can assist in a best effort practice since our support team is intended as a break fix support.

Conceptual Flow: “Drive Space Remediation Before Alerting”

Monitor: Alert triggers when volume hits 95% usage.
Remediate: Automatically run PowerShell to clean specific folders.
Wait & Re-evaluate:
- If space drops below threshold, reset alert.
- If space still over threshold after X mins, escalate/send alert.

Suggestion:

Step 1: Create the Drive Space Alert (Silent Trigger + Script)

Trigger Condition:
- Alert when:
 mathematica
 CopyEdit
 Volume Percent Used >= 95%AND Volume Type = Fixed Disk
Trigger Actions:
1. Execute External Program (or use PowerShell directly):
 - Script deletes data from specific directories on the affected node.
 - Use ${N=SwisEntity;M=Node.Caption} and other variables for dynamic execution.
 ps1
 CopyEdit
 # Sample cleanup.ps1Remove-Item -Path "C:\Temp\*" -Recurse -Force -ErrorAction SilentlyContinue
 If using “Execute Program”:
 cmd
 CopyEdit
 powershell.exe -File "D:\Scripts\cleanup.ps1" -ComputerName ${N=SwisEntity;M=Node.Caption}
 Make sure the script runs remotely or via agent, or push via SolarWinds agent-based action.
2. Delay Further Actions: Set "Condition Must Exist for X Minutes", e.g., 5 minutes.
Reset Condition:
- Volume Percent Used drops below 95% (e.g., to 90%).
Do not send notification in this alert — this is your self-healing trigger.

Step 2: Create a Follow-up Alert (If Space Still Over Threshold)

Trigger Condition:
- Same as first alert, but:
 sql
 CopyEdit
 Alert only if condition exists for >5 minutes
Alert Actions:
- Send email/webhook notification
- Raise incident ticket (e.g., ServiceNow)
- Escalate to on-call/NOC

Key Settings & Options

Setting	Recommendation
"Condition must exist for"	Use this to wait post-cleanup before escalating
Execute Script On Node	Use agent or credentials that allow remote PowerShell
Reset Action	Optional: log that self-healing worked
Custom Property (Optional)	Tag nodes that are eligible for auto-remediation

️ Tips

Use SolarWinds Agent on Windows nodes to allow “Run a program/script on node”.
Test the cleanup script thoroughly with logging before tying to alert.
Secure your PowerShell scripts—especially if they delete anything.
Log actions for auditing (e.g., write to file or event log).

Example Workflow Summary

Step	Action
1	Disk >95% → Silent alert triggers cleanup script
2	Script runs → deletes files
3	Alert waits 5 min
4a	If disk <95% → alert auto-resets (no escalation)
4b	If disk still >95% → second alert fires with notification/escalation

PowerShell script

That can help clean up directories and log actions.

Script Goal

Delete files from:
- C:\Temp
- C:\Windows\Temp
- C:\Logs\Old
Log what was deleted and any errors
Output free space on C: drive before and after cleanup
Exit with exit code 0 if cleanup successful (space < 95%)
Exit with exit code 1 if still over threshold

PowerShell Script: `cleanup.ps1`

# cleanup.ps1 # Description: Frees up disk space and exits with a code that can be used to determine alert escalation. $LogFile = "C:\cleanup_log.txt" $DriveLetter = "C:" $ThresholdPercent = 95$FoldersToClean = @("C:\Temp", "C:\Windows\Temp", "C:\Logs\Old") $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" Function Log {   param ([string]$message)   "$timestamp $message" | Out-File -FilePath $LogFile -Append } Function Get-UsedPercent {   $drive = Get-WmiObject Win32_LogicalDisk -Filter "DeviceID='$DriveLetter'"   if ($drive -eq $null) {     Log "ERROR: Could not retrieve drive info." return 100   }   $usedPercent = (($drive.Size - $drive.FreeSpace) / $drive.Size) * 100 return [math]::Round($usedPercent, 2) } # Log start Log "Starting cleanup script..." # Check initial disk usage $initialUsage = Get-UsedPercent Log "Initial disk usage: $initialUsage%" # Begin cleaning foreach ($folder in $FoldersToClean) {     if (Test-Path $folder) {       try {         Get-ChildItem -Path $folder -Recurse -Force -ErrorAction SilentlyContinue |         Remove-Item -Recurse -Force -ErrorAction SilentlyContinue         Log "Cleaned contents of: $folder"       } catch {         Log "ERROR cleaning $folder: $_"       }   } else {     Log "Folder not found: $folder"   } } Start-Sleep -Seconds 10 # Give the system a moment to recalculate space # Check final disk usage $finalUsage = Get-UsedPercent Log "Final disk usage: $finalUsage%" if ($finalUsage -ge $ThresholdPercent) {   Log "Disk usage still high. Cleanup may not have been sufficient."   exit 1 } else {   Log "Disk usage within acceptable range. No alert needed."   exit 0 }

Use case:

Use the exit code in your SolarWinds alert action to determine:
- If exit 0: Do nothing further (suppress alert)
- If exit 1: Fire second alert or escalate to ticket/email

Hope this helps.