WPM: Probe process has failed with exit code 1

We have been running WPM transactions for years, recently we have been seeing our players generate this error :Probe process has failed with exit code 1. We have engaged support, they have had us repair the client, turn on additional logging and numerous other ideas but they have not been able to figure out what is going on. We just rebuilt the server and add the transactions back on it and life is good. We thought this would be a onetime issue but now we have seen it happen on multiple player nodes. We have 12 player nodes and we have now seen this issue on 4 out of the 12 in the last 2 months. Prior to this we have never seen this issue (we have been using WPM for about 6 years). We even tried a restore on one, it worked for a couple days and then had issues again. I didn't see anything being installed on that device at the time of failure either. Thoughts?

Find more posts tagged with

Accepted answers

All comments

michal.kalina

'Probe process has failed with exit code 1' is a generic error and it can mean anything bad that can happen with the probe process over its runtime lifetime. To find out more about what is causing it I would enable verbose logs on those WPM Players on which the issue starts occurring and then I would analyze AgentWorker_*.log.# files to learn why it keeps failing.

Another option - on one of your players reduce the number of transactions a bit and see if the issue is reduced or does not appear any longer.

Additionally, check how CPU/RAM usage looks like on WPM Player locations when the issue occurs next time - does it differ from the CPU/RAM usage on that location when the issue does not occur? Do you observe any hanging WPM Browser.exe/chromedriver.exe/probe.exe process in TaskManger on that location?

Do you have latest updates?

And lastly, have you tried to configure --disable-gpu flag in AgestSettings.dat file?

As a last resort, I would contact SW support to assist you with the investigation.

PatrickRG

I just had a second reoccurrence of the same issue you describe on the same APE. What version are you running? Support had me stop the WPM Player and rename the C:\ProgramData\SolarWinds\Logs\SEUM\Audit folder to Audit.old. That fixed my failed transactions temporarily, but it seems like those audit logs will fill up and cause the issue again in the future.

bmoline

renaming the folder works most of the time. (the first server we had with this issue, renaming the audit folder did not fix the issue and we just rebuilt it). We are running into the same issue with the audit logs, 2Million+ files on some. We also saw one server that the c:\user\SEUM accounts were using 12 GB of space each and not releasing it. I was able to delete the accounts, folders and let them recreate. We are on the latest, 2024.4.0. I still have a case open on this.

PatrickRG

Thanks for the info. I forgot to mention I'm on 2024.2.0 still. The issue just cropped up for me about a month ago for the first time and today for a second time.

bmoline

I got an update from our Dev Team about our issue with the Audit logs.
The issue is now considered a bug and tracked under OO-40267.

Here's the workaround provided:

The issue with growing the content of the Audit folder is known - by design, it is not being purged - here is a list of file prefixes that we are taking care of
@AgentWorker_*,

@AgentWorkerlegacy_*,

@AgentProcessLauncher_*,

@AgentRDPClient_*,

@chromeDriver_*

We know that it can cause some issues when you want to collect logs manually and, for example, zip the SEUM folder, which might take a long time.

Lastly, you can delete the content of that file as you wish - it has no diagnostic value for us. But it can be used by some Security Technician that is analyzing when WPM Player started its sub-processes.

If the customer wants to clean it up regularly, he can create a task in the Task Scheduler that runs, for example, a PowerShell script that removes files from the Audit folder older than a certain number of days.

Here is one of many examples of how this can be done: https://pureinfotech.com/delete-files-older-than-days-windows-10/

You can free up space by deleting files that are older than a certain number of days in any folder—here's how.

https://pureinfotech.com/delete-files-older-than-days-powershell/

To delete files older than X number of days on Windows 10, open PowerShell and run "Get-ChildItem" with "Remove-Item" and these options.

PatrickRG

thanks @bmoline . I implemented a scheduled purge this morning. It seems like something changed with WPM recently though; I suspect a new log file is not supposed to be created every time the player runs which is what it appears to be doing currently. I ran for years without issues but we filled the SEUM\Audit logs in about 30 days after I purged them completely. I'll be keeping an eye out for their fix.

bmoline

Same, we have been using WPM for many years, we have hundreds of transactions and dozens of players, we just started seeing it earlier this year as well.