This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.

You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

How to Do URL Queries Using CURL

sumairian over 10 years ago

Howdy, folks! I'm pretty new to SolarWinds IPAM. And while I've been scripting for a while, I feel I'm still relatively new to `curl'. At any rate, here's my premise: if I can pass a URL successfully to my browser and get good data, I should be able to pass the same URL to `curl' to get text-based output. Unfortunately, I've been hitting my head up against a wall, and I'm out of tricks at this point.

Why am I doing this? Ultimately, I want to get IP Assignment History. But I'm taking baby steps, and here's what I'm sending to my browser right now. It's an IPAM query equivalent to a basic `nslookup', viz., "solarwinds/.../search.aspx Note that this is an internal address, and that I actually get a hostname back when I send this to my browser.

However, when I try to lookup the same URL from the command line, I don’t get the expected output. Instead, I get garbage.

Here's my simplified `curl' statement:

$ curl -L -u USER solarwinds/.../search.aspx

Enter host password for user 'USER':

Full disclosure: here’s my actual script…

#!/usr/bin/sh

# Dynamic Variables

# -----------------

IP_LOOKUP="192.168.11.10"

URL="solarwinds/.../search.aspx

BROWSER='Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)'

COOKIE='XLI5P8GU.txt'

# Get IP History

# -----------------

curl -L -c ${COOKIE} -u USER -A "${BROWSER}" "${URL}" | \

sed 's/<[^>]*>//g'

…and here’s what happens when I run it: I get nothing relevant after stripping out the HTML

# ./curl_solarwinds.sh

Enter host password for user 'USER':

<I enter this manually>

SW.Core.Loader._cbLoaded('jquery');

SW.Core.Date._init(0,-25200000);

<white space removed>

window.location = 'Login.aspx';

Your ninja-kungfu-help is appreciated!

Sumair

Top Replies

jangliss over 10 years ago +2

The problem you have is that your cURL call doesn't have a valid session. As such your request is being redirected back to the login page (which is the junk you see in the output). If you go one step…

0 jangliss over 10 years ago

The problem you have is that your cURL call doesn't have a valid session. As such your request is being redirected back to the login page (which is the junk you see in the output). If you go one step further and dump the headers as well, you'll see SolarWinds is sending an HTTP Location: header, and redirecting the client. So for example:

curl -L -c ${COOKIE} -D "header_out.txt" -u USER -A "${BROWSER}" "${URL}"

Now for the headers:

HTTP/1.1 302 Found

Cache-Control: private

Content-Type: text/html; charset=utf-8

Location: /Orion/Login.aspx?ReturnUrl=%2fOrion%2fIPAM%2fsearch.aspx%3fq%3d10.5.21.15&q=10.5.21.15

Server: Microsoft-IIS/7.5

X-AspNet-Version: 4.0.30319

X-Powered-By: ASP.NET

Date: Fri, 28 Feb 2014 20:55:50 GMT

Content-Length: 208

As you can see, a 302 and Location headers are redirecting us back to the login page.

So there are 3 solutions to this. The first is to capture the login form, find the fields, use the post options, capture more cookies, then go back to the search page using the newly established session data. This is a lot of fiddling around, and might not be worth it. The second is establishing a browser session, and copying the cookie data from that, and saving it to a file, and using it as such:

curl -L -b "cookie.txt" -u USER -A "${BROWSER}" "${URL}"

This should work as the session is already established, assuming you do not allow the sessions to expire for this user.

The third, and probably easiest is dependent on how you have curl compiled. This uses the --ntlm option (assuming you are using Windows logins, and have NTLM enabled in IIS). The change to your command line is pretty small:

curl -L -c ${COOKIE} --ntlm -u USER -A "${BROWSER}" "${URL}"

When I executed this, it returns all the data. One caveat is that if the IP matches extra data, such as in my case 10.5.21.150-159 were also returned, that is in your data too.

I'm not sure how well your sed command will work on the results because the actual data is inside a javascript array object at the bottom of the page, which then self-injects itself into the table, but I'm sure once you have the data you can play around with it.