1 Reply Latest reply on Feb 28, 2014 3:25 PM by Jonathan Angliss

    How to Do URL Queries Using CURL


      Howdy, folks!  I'm pretty new to SolarWinds IPAM.  And while I've been scripting for a while, I feel I'm still relatively new to `curl'.  At any rate, here's my premise:  if I can pass a URL successfully to my browser and get good data, I should be able to pass the same URL to `curl' to get text-based output.  Unfortunately, I've been hitting my head up against a wall, and I'm out of tricks at this point. 

      Why am I doing this?  Ultimately, I want to get IP Assignment History.  But I'm taking baby steps, and here's what I'm sending to my browser right now.  It's an IPAM query equivalent to a basic `nslookup', viz., "http://solarwinds/Orion/IPAM/search.aspx?q="  Note that this is an internal address, and that I actually get a hostname back when I send this to my browser.


      However, when I try to lookup the same URL from the command line, I don’t get the expected output.  Instead, I get garbage. 


      Here's my simplified `curl' statement:

      $ curl -L -u USER http://solarwinds/Orion/IPAM/search.aspx?q=

      Enter host password for user 'USER':




      Full disclosure:  here’s my actual script…


      # Dynamic Variables

      # -----------------



      BROWSER='Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)'



      # Get IP History

      # -----------------

      curl -L -c ${COOKIE} -u USER -A "${BROWSER}" "${URL}" | \

      sed 's/<[^>]*>//g'


      …and here’s what happens when I run it:  I get nothing relevant after stripping out the HTML

      # ./curl_solarwinds.sh

      Enter host password for user 'USER':

      <I enter this manually>

      (function(){var de=$(document.documentElement); de.addClass('sw-is-locale-en'); $.each(jQuery.browser,function(k,v){if(v===true){ de.addClass('sw-is-'+k); de.addClass('sw-is-'+k+'-'+parseInt(jQuery.browser.version)); }}); })();





      <white space removed>




          window.location = 'Login.aspx';



      Your ninja-kungfu-help is appreciated!



        • Re: How to Do URL Queries Using CURL
          Jonathan Angliss

          The problem you have is that your cURL call doesn't have a valid session. As such your request is being redirected back to the login page (which is the junk you see in the output).  If you go one step further and dump the headers as well, you'll see SolarWinds is sending an HTTP Location: header, and redirecting the client.  So for example:


          curl -L -c ${COOKIE} -D "header_out.txt" -u USER -A "${BROWSER}" "${URL}"


          Now for the headers:


          HTTP/1.1 302 Found

          Cache-Control: private

          Content-Type: text/html; charset=utf-8

          Location: /Orion/Login.aspx?ReturnUrl=%2fOrion%2fIPAM%2fsearch.aspx%3fq%3d10.5.21.15&q=

          Server: Microsoft-IIS/7.5

          X-AspNet-Version: 4.0.30319

          X-Powered-By: ASP.NET

          Date: Fri, 28 Feb 2014 20:55:50 GMT

          Content-Length: 208


          As you can see, a 302 and Location headers are redirecting us back to the login page.


          So there are 3 solutions to this.  The first is to capture the login form, find the fields, use the post options, capture more cookies, then go back to the search page using the newly established session data.  This is a lot of fiddling around, and might not be worth it.  The second is establishing a browser session, and copying the cookie data from that, and saving it to a file, and using it as such:


          curl -L -b "cookie.txt" -u USER -A "${BROWSER}" "${URL}"


          This should work as the session is already established, assuming you do not allow the sessions to expire for this user.


          The third, and probably easiest is dependent on how you have curl compiled.  This uses the --ntlm option (assuming you are using Windows logins, and have NTLM enabled in IIS).  The change to your command line is pretty small:


          curl -L -c ${COOKIE} --ntlm -u USER -A "${BROWSER}" "${URL}"


          When I executed this, it returns all the data.  One caveat is that if the IP matches extra data, such as in my case were also returned, that is in your data too.


          I'm not sure how well your sed command will work on the results because the actual data is inside a javascript array object at the bottom of the page, which then self-injects itself into the table, but I'm sure once you have the data you can play around with it.

          1 of 1 people found this helpful