Can SolarWinds SAM or WPM do web scraping?

I am looking to see if either the SAM or WPM tools can do some form of web scraping.

For instance, load a web page, look for a text value on the web page and capture (not as an image) the value after that text value.

I know SAM can look for a text value on a web page and if found show up or down and WPM can do XY captures but I may trying to squeeze more out these tools, if possible.

A simple example would be to navigate to a node detail page and look for 'Last 30 Days' under Availability Statistics and capture that value. (This is just an example for discussion, I know that these values are in the database)

Find more posts tagged with

Accepted answers

All comments

yaquaholic

It's at times like these we could do with Windows versions of wget, sed and awk...

Though wget has a Powershell alias to Invoke-WebRequest, and is also very capable of searching for and dealing with strings...

Here's a few lines that might help you get started:

$uri = 'http://thwack.solarwinds.com'

$pattern = 'THWACK MONTHLY MISSION'

$found = (Invoke-WebRequest -Uri $uri ).Content

$found.ToString() -split"[`r`n]" | Select-String $pattern

chad.every

+1 to what yaquaholic said. I've used SAM and powershell to scrape info off websites that didn't work with the HTTP component monitor or with WPM.

FormerMember

yaquaholic, chad.every,

Can you provide me more details on how you implemented this?

I created a new template and used the 'Windows PowerShell Monitor" and pasted in the content and test but it comes back Output Result - Not defined

Not sure what I am missing

yaquaholic

Hi Sotheris,

The PowerShell monitors need to return their results in a specific format to be accepted, that being that the results are wrapped with a "Satistic: <value>" and an optional "Message: <string>" formatting.

Have look in your Program Files (x86)\SolarWinds\Orion\APM\SampleScriptMonitors\PowerShellScripts, where you can find two working examples, "Multi Value - Script Example (Process Info).ps1" is a good example, showing how you can assign multiple return values, using the "Message.<name>" and "Statisitc.<name>" format, which allows up to ten results per script. But if your script returns only one value you can just use "Statistic:<value>".

Have a play with it, see how you get on. There are plenty of resources on Thwack and the Customer portal that will guide you, such as - Creating a Windows PowerShell monitor - SolarWinds Worldwide, LLC. Help and Support

Shout, if you are still stuck

FormerMember

Well, I did add the message and statistics part and tried also to put the supplied code into a powershell file and run it but it throws errors so still digging into why that is.

yaquaholic

Yes, they can be rather troublesome at times.

Especially on the first few attempts

First I'd use PowerShell ISE, to ensure that the script will run and returns the correct data to the prompt, then wrap in the required Statistic and optional Message strings.

Once i am happy with my code and the returned results, I'd then take into SolarWinds, as a PowerShell monitor, in my application template.

These PowerShell monitors can be run in two ways, either run locally on the Orion server or remotely on the monitored host.

If it's the latter you need to ensure that you can run remote PowerShell and permissions, but I'd imagine you're just using a locally run script, if your Orion server has access to the URLs you are scraping.

Set where the script is to be run, by default it is locally on the Orion server, and then use the edit script window to test your script.

Keep it simple at first, go for one Statistic and one Message to be returned, you can build on it, once you have the script working.

If you're stuck, feel free to stick the script up and I'll (we'll - nods to other Thwackers) have a look for you.

yaquaholic

@sotheris how did you get on?

#curious #slowday