No, you haven't entered a multidimensional time warp. Nor are you having a 90's flashback. While the industry hype cycle is primarily focused on hot new trends like hybrid IT, SaaS, and containers, there lurks an unsung hero in the darkest dwellings of many of today's most established organizations. It oftentimes doesn't get the attention or appreciation it deserves, because to most, its existence is completely transparent. It sits there in the corner, just plugging away, day after day, hour after countless hour, without complaint or need for recognition. Yet these systems remain at the very core of the business, handling the most critical transactions. From maintaining patients medical records, keeping all your banking transactions in order, to running some of today's largest companies CRM and ERP applications, AIX is still very much around us every day, touching our lives in ways you probably haven't even considered.

 

For as important as these systems remain even today, the monitoring of their performance and application health is far too often overlooked or completely forgotten. Perhaps it's because these workhorses were built to last and seldom fail at their important duties, making them fall into the dangerous category of out-of-sight-out-of-mind. More likely, however, is that these systems have traditionally been extremely difficult to gain visibility into using modern day multi-vendor monitoring solutions. That may be because a long time ago, IBM seemingly stole a page out of Sony's playbook of market dominance, which had propelled their proprietary Betamax and MiniDisk formats into the iPhone like successes of their day. Oh, wait. That's not what happened! That's not what happened at all!!

 

Unfortunately, despite strict and very well defined industry standards which would govern how key operating system metrics should be exposed, and allowing third-party monitoring solutions to provide necessary insight into their health and performance, IBM decided that standards didn't necessarily apply to them. This decision has historically made monitoring AIX systems challenging for both their customers, as well as, 3rd parties seeking to provide a monitoring solution for those organizations most critical systems. Compounding this problem is the fact that the few monitoring solutions available to those customers have traditionally been wildly complex, difficult to deploy and configure, and even more challenging to maintain.  A new solution was needed. One which could bring with it unexpected simplicity, where none existed before. Its time has come, and that time is now.

 

 

AIX Agent Deployment

 

As one would expect from SolarWinds, deployment of the AIX agent is a simple turnkey affair, no different than deploying Agents to other operating systems, such as Windows or Linux. That's right, deploying an Agent to AIX is just as simple as it is for Windows and you don't need to be an expert in AIX. In fact, you don't even need any experience using AIX to be successful monitoring these systems with Server & Application Monitor (SAM) 6.6. If you can add a Node in Orion, then you, too, can monitor your AIX 7.1 and 7.2 systems.

 

Add Node Wizard - Push Deployment

 

To begin, navigate to [Settings -> All Settings -> Add Node], enter the IP address or fully qualified hostname of the AIX host you'd like managed in the "Polling Hostname or IP Address" field and select the "Windows & Unix/Linux Servers: Agent" radio button from the available "Polling Method" options. Next, enter the credentials that will be used to both connect to the AIX host and install the agent software on the 'Unix/Linux tab. The credentials provided here should have 'root' or equivalent level permissions. Note that the credentials provided here are used only for initial deployment of the agent. Future password changes of the account credentials provided here will have no impact on the agent once it is deployed. Alternatively, if you authenticate to your AIX host via SSH using a client certificate rather than a username and password, click the 'Certificate Credential' radio button and upload your certificate in PEM format through the Orion web interface. This certificate will then be used to authenticate to the AIX host for the purpose of installing the Agent.

 

You can also optionally add SNMP credentials to the Agent if SNMP has already been configured properly on the AIX system. Rest assured, though, that this isn't needed and is used only if you're wanting to utilize SAM's SNMP Component Monitors against the AIX system. Configuring this option will also populate the 'Location' and 'Contact' fields located on the 'Node Details' resource if those values have been properly populated in your SNMP configuration. Everything else will be polled natively through the AIX Agent with zero reliance upon SNMP.

 

 

Once you've entered your AIX credentials, click the 'Next' button at the bottom of the page. The Agent will then be deployed to the AIX host using a combination of SSH and SFTP requiring TCP port 22 be open from the Orion server (or additional polling engine) to the AIX endpoint you wish to manage for push deployment to function properly.

 

Install Agent PromptInstall Agent Progress IndicatorList Resources

 

Manual - Pull Deployment

 

In some scenarios, it may not be possible for the Orion server to push the agent to the AIX host over SSH. This is not uncommon when the host you wish to manage resides behind a NAT or access control lists restrict access to the AIX system via SSH from the network segment where the Orion server resides.  While firewall policy changes, port forwarding, or one-to-one address translations could be made to facilitate push deployment of the agent, in many cases, it may be far easier to perform a manual deployment of the agent to those hosts.

 

The Agent package can be downloaded from the Orion web interface to the AIX host by going to [Settings -> All Settings -> Agent Settings -> Download Agent Software] and selecting "Unix/Linux" from the options provided and clicking "Next".

 

Download Agent Software - Machine Type
Download Agent Software - Deployment Method

 

In the following step of the Wizard select "Manual Install" and click "Next". Finally, in the third and final step of the wizard is where you will select 'IBM AIX 7.x' from the 'Distribution' drop down. Here you can also configure any advanced options the agent will use when it is installed, such as which polling engine the Agent should be associated with in Agent Initiated (Active) mode, or the listening port the Agent will use when running in Server Initiated (Passive) mode. Additionally, you can also specify a proxy server the Agent should use to communicate to the Orion server or Additional Polling Engine in Agent Initiated (Active) mode. If you're deploying in an environment where proxy servers are used, fret not. The Agent's proxy configuration fully supports the use of authenticated proxies.

 

 

After selecting all the appropriate configuration options, click the "Generate Command" button at the bottom of the page. This will generate a dynamic installation command based upon the settings chosen above, which can then be copied and pasted into an SSH or X-Windows session on the AIX host. The AIX machine will then download and install the appropriate agent software from the Orion server using those pre-configured options.

 

Copy Generated Agent Installation CommandPaste Command into SSH TerminalAgent Installation Success

 

As soon as the Agent is registered with the Orion server, select your newly added agent node and click "Choose Resources" from the 'Manage Agents' view to select items on the node you would like to monitor.

 

 

Agent Advantages

 

So what's so great about Agents anyway? What's wrong with using the tried and true agentless methods for monitoring AIX hosts, like SNMP?

 

Encryption

 

Well, as anyone who has the misfortune of using SNMP to monitor their AIX hosts can tell you, it's not all sunshine and lollipops, starting with configuring SNMP. Most environments today have strict security standards which mandate the use of encryption for virtually all network communication. While configuring SNMP v1/v2 on AIX isn't especially difficult for an experienced AIX administrator, neither of those versions of SNMP utilizes encryption. That would necessitate that users utilize SNMPv3, which comparatively speaking, practically requires users obtain a Ph.D. from Big Blue University in AIX to properly enable and configure.  By comparison, the Orion AIX Agent natively utilizes highly secure 2048 bit TLS encryption for all network communication.

 

Visibility

 

IBM's proprietary SNMP daemon leaves much to be desired when compared to other standard based SNMP daemons running on alternative operating systems. Chief among the complaints I hear regularly is that IBM's SNMP daemon doesn't support important standard MIBs, such as the HOST-RESOURCES-MIB which exposes key pieces of information regarding running processes on the server and their respective resource consumption. This remains the primary reason why so many customers have chosen to replace IBM's proprietary SNMP daemon with NET-SNMP.  SImilar to NET-SNMP, though, are issues of reflecting critical metrics accurately, such as memory utilization. It seems odd that something so basic would present so many challenges and be pervasive across both Linux and AIX when monitored via SNMP.

 

Reliability

 

Like all Agents in Orion, the AIX Agent runs independently of the Orion server. This means the Agent continues to monitor the host where it's installed, even if the Orion server is down, or otherwise inaccessible due to a network outage. Once connectivity is restored or the Orion server is brought back online, the data collected by the AIX agent is then uploaded to the Orion server, filling gaps in the historical time series charts that would have otherwise existed if that node was being monitored via SNMP. This ensures that availability reporting is accurate, even if the server running Orion experiences a catastrophic failure.

 

Reachability

 

In today's highly secure and heavily firewalled environments which are riddled with network obstacles such as network address translation, port address translation, access control lists, and proxies, it's sometimes amazing that anything works at all. More and more the things that need to be monitored are oftentimes the most difficult to monitor. With the AIX agent, overcoming these obstacles is a snap, allowing users to monitor their AIX systems regardless of where they might be located in the environment. Have your Orion system deployed in the Cloud and running in Amazon's AWS or Microsoft's Azure? Not a problem. Deploy the Agent in Agent Initiated (Active) Mode and forget about VPN tunnels or 1:1 NAT mappings. Does all traffic leaving the network go through a proxy server? No problem. The Agent natively supports the use of authenticated proxies to access the Orion server, while conversely, Agent communication within Orion can be configured to utilize a proxy server to reach an Agent that might not be accessed directly. These are possibilities you previously could only dream about when using SNMP.

 

 

AIX Agent Exclusive Features

 

There have been several Orion features released throughout the years which had previously only been available for nodes running other operating systems, such as Linux or Windows. AIX had largely been left out in the cold. That is, until today.

 

Network Interface Monitoring

 

In Server & Application Monitor 6.6, network interfaces on your AIX server can now be monitored without needing Network Performance Monitor installed. This functionality is available exclusively through the AIX Agent and does not count against your SAM component monitor usage or NPM element license count, in the event you also have NPM installed. That means this functionality is provided essentially free and potentially even allows you to free up some of those valuable NPM element licenses for other nodes in your environment.

 

Volume Performance Monitoring

 

Today, storage is the leading cause of server and application performance issues. Having visibility into storage volume performance, such as disk reads/writes per second, and queued disk I/O from within the Orion web console alongside other key performance indicators, allows you to isolate where performance bottlenecks are occurring on your server and which applications are affected. With the AIX Agent, you now have visibility into the storage volume performance, similar to those that you've grown accustomed to on your Windows and Linux volumes.

Total Disk IOPSDisk Queue Length

 

 

Real-Time Process Explorer

 

When applications aren't running right, inevitably there's a culprit. It may be the processes that make up the application you're already monitoring, or it might be with those you aren't. The Real-Time process explorer provides you visibility into all processes and daemons running on your AIX server, along with their respective resource utilization. It's like your web-based command center where you can quickly determine which processes are running amuck. No more firing up your SSH client, logging in and running 'topas' to troubleshoot application issues. Now you can do it all from the comfort of your web browser. Spot a runaway process or one that's leaking memory like a sieve? You can also terminate those processes directly within that same web interface. Simply select the process(s) you want to 'kill' and click 'End Process'. Voila! It's just that easy.

 

You can also now select processes you wish to monitor on your AIX server directly through the Real-TIme Process Explorer. To do so, simply select the process you're interested in monitoring and click 'Start Monitoring'. You'll then be walked through SAM's application template wizard where you can choose to add this process to one of your existing Application Templates or create a new one.

 

 

 

Reboot Management Action

 

If you find yourself in a situation where terminating processes alone does not resolve your application issue, there's always the tried and true reboot. While usually the option of absolute last resort, it's comforting to have it easily at hand when and if you've exhausted all other options. Simply click the 'Reboot' button in the 'Management' resource on the 'Node Details' view and you'll be prompted to confirm that you really mean business.

 

 

Application Component Monitors

 

Last, and unquestionably most important, are the wide array of various SAM Application Component Monitors supported by the AIX Agent. From these components, you can create templates to monitor virtually any application, commercial, open source, or homegrown.

 

AIX Agent Supported SAM Component Monitor Types

Directory Size Monitor

File Count Monitor

HTTPS Monitor

ODBC User Experience Monitor

DNS User Experience Monitor

File Existence Monitor

JMX Component Monitor

Process Monitor

File Age Monitor

File Size Monitor

Linux/Unix Script Monitor

SOAP Monitor

File Change MonitorHTTP MonitorNagios Script Monitor

SNMP Component Monitor

TCP Port Monitor

Tomcat Server Monitor

animelov

SWQL Walkthrough

Posted by animelov Employee Mar 13, 2018

Hi, all!  And welcome back to our discussion on the SDK and all things SWQL. So, at this point, we’ve briefly introduced SWQL, but today we’re going to get down to building queries with SWQL and talk about what you can make with them.

 

But first, let’s discuss why this is important. The Orion® Platform does an excellent job of giving you a single-pane-of-glass view for all of your products, while giving you the freedom to pick and choose which modules you wish to purchase. Because of this, the back-end database is fairly segregated, save for a handful of canned widgets and reports. For database experts, this would be done by waving a magic wand and using the correct incantations of “Inner join!” or “Outer Join!”  For the rest of this, SWQL can do the trick!

 

Now, what this is NOT going to be is a general guide on structured query language (SQL). This does require some level of knowledge of how to construct a basic query, but don’t be scared off just yet.  SWQL, let alone SQL, is not something I knew before I started, but I picked up quite easily.

 

Also, before we begin, if you haven’t picked up the latest SWQL studio, I highly recommend you do so and check out our most recent post here. This can be installed anywhere that has an Orion server connection, including your laptop.

 

Now that that is out of the way, let’s get to the meat of the subject. SWQL, in and of itself, is very similar to SQL, but with a few restrictions. For example, SWQL and its studio is read-only. There are SWIS calls you can make with it for API/PowerShell®, but we’ll get into that in a later post. For a quick rundown, check out this post.

 

If I wanted to see a list of applications that I’m monitoring on my SAM instance, I could write a query that looks like this:

select Displayname from Orion.APM.Application

 

And I go to Query à Execute (or F5),

 

that will get me an output that looks like this:

 

Pretty simple, right?  Let’s look and see what this does, though:

 

 

“select” and “from” should be self-explanatory if you know a little SQL. “select” means we want to retrieve/read information, and “from” is stating which table we’re getting that information from.  Orion.APM.Application is the table name (so, we’re getting information “from” “Orion.APM.Application”), and “DisplayName” is the title of the column we’re getting that information from.  Now, where did we get that column name from?  If you look at SWQL studio and find the table name and expand on it, you’ll get this:

 

 

Those icons with the blue icons next to them? Those are the columns we can pick from. For the other icons, check out our previous post here.

 

Let’s add some more to that query. If we want to see the status of the applications (up/down/warning/etc.), we can just add the status to the query, like so:

select Displayname, Status from Orion.APM.Application

 

This will give us:

 

More info on the status values can be found here, but, 1 is up, 2 is down, etc.

 

Now, what if we wanted to select ALL of the columns in this table to see what we get. Unfortunately, this is one of the first things that differs from SQL to SWQL that you cannot wildcard with *.  In other words:

 

select * from <tablename> does NOT work!!!

 

If you want all columns, you’ll have to select each one of them, separated by a comma. Luckily, SWQL will do this for you. If you right-click on the table name, you have the option of creating a general select statement:

That will generate a query for you with EVERY column possible for that table.

 

Pretty neat, right? Now, let’s get to the fun part of SWQL. One of the attributes of SWQL over SQL is its ability to automatically link related tables. If you are familiar with SQL, this is where things would normally get hairy with inner/outer join statements. Here we make it easier.

 

Let’s use our application example again. Having the list of applications is great, but to me, it’s nothing unless I know which node it is tied to. There is a node id in that application table, but it returns a number, which means nothing to me.  Remember those chain link icons from earlier?  Those are linked tables, and if you look, there is a link to the Orion.Nodes table:

 

 

To get the linkage to work, we first need to give the application table an alias. To do so, I just need to give it a name after the table declaration.  So, let’s call it “OAA,” which is short for Orion APM Application. Note: you can name it anything EXCEPT for a reserved word, like “select” or “from.”

select Displayname, Status from Orion.APM.Application OAA

 

Now, we need to make sure our columns are referenced to OAA by adding that to the beginning of the column names:

select OAA.Displayname, OAA.Status from Orion.APM.Application OAA

 

Finally we add the node name.  We can do this with the linked table from earlier by using dot notation.  In other words, if I write in “OAA.Node.”, I’m now allowed to pick any column from the node table, including the name of the node (or “caption”).  Now my query looks like this:

select OAA.Displayname, OAA.Status, OAA.Node.Caption from Orion.APM.Application OAA

 

And now, this is my output:

 

This is where things get interesting. Remember how I said that we can tie multiple products together?  The AppStack dashboard with Virtualization Manager and Storage Resource Monitor is extremely powerful, especially in terms of reporting. SWQL can help us with that.

 

If we keep going with our Application example above, we can continue tying information together from other tables.  So far, we’ve linked the nodes table, but let’s see what ESX host these belong to. From the Applications table, there isn’t a link to any of the VIM tables:

 

But it is related to the “Nodes” table, and if we look at the Nodes table:

Then we go to the Virtual Machines table and from Virtual Machines table…

… there’s the Hosts table!  So, let’s link that to our query using dot notation:

select OAA.Displayname, OAA.Status, OAA.Node.Caption, OAA.Node.VirtualMachine.Host.HostName from Orion.APM.Application OAA

 

Note, the host names that are NULL refer to machines that are not monitored via Virtualization Manager; they do not have a host.

 

That’s it for now. Later, we’re going to learn some more tricks for formatting with SWQL, and then how to apply this to Orion®. Stayed tuned for the next one!

We are happy to announce the release of SolarWinds® Traceroute NG, a standalone free tool that finds network paths and measures their performance.

The original traceroute is one of the world’s most popular network troubleshooting tools but it works poorly in today’s networks. You can read about its shortfalls in this whitepaper.

SolarWinds® fixed these shortfalls with NetPath, a feature of NPM.  People love NetPath but there are two problems.  First, NetPath takes a couple minutes to find all possible paths in complex networks, much longer than a quick tool like traditional traceroute. Second, most people don’t own SolarWinds® NPM and so don’t have access to NetPath.

Traceroute is too important of a tool to allow it to languish.  That’s why we’ve taken what we’ve learned with NetPath and fixed traceroute.  We call it Traceroute NG.

Traceroute NG is a super fast way to get accurate performance results for a network path in a text format that’s easy to share.

Compared to traceroute, TracerouteNG is:

  • Super-fast
  • Rarely blocked by firewalls
  • More accurate, thanks to path control
  • Updates latency/loss continuously
  • Detects path changes
  • TCP or ICMP

 

You can download Traceroute NG here and launch the tool by double-clicking the traceng.exe.

You’ll be presented with a help screen and the application will wait for your input. Type the domain name to start a trace.

 

You can also launch the free tool from Windows command prompt:

traceng www.google.com

Let’s look at some results.

 

Scenario 1: Endpoint is blocking TCP port

We all know that HTTP uses TCP 80 by default. What would traceroute show you if someone blocks that port on a firewall or webserver?

All good, it’s not the network. You know it’s not your issue. But what is the issue?  That’s where Traceroute NG will help:

Traceroute NG can mimic TCP application traffic, so packets are treated as the application traffic is. In this case it detected that port TCP 80 on the destination webserver is closed. You know, it’s not the network. But you can be more precise and tell your sysadmin to enable this port on his webserver.

 

Scenario #2: Network path change

To illustrate this scenario, I have created a simple network using GNS3.

 

I also have a loopback adapter configured, to point all IPv6 traffic to this lab:

 

I’d like to trace from my machine in Cloud1 to the PC (fc90::3). If the OSPF routing works, I should go through routers R1, R7 and R3. Traceroute confirms:

 

Traceroute NG as well:

 

What if I do maintenance on router R7? Will traceroute tell me, when router R7 becomes unavailable and detect the new path? No. It runs once and then you need to run it again. Manually.

With Traceroute NG, detecting a change is simple. You can tell Traceroute NG to warn you if the path changes and optionally log the output. An example command would be: -a warn -l -p 23 fc90::3

And this is the result:

 

So you know that your router is down and once you hit enter, Traceroute NG will show you the new path. In the GNS3 lab we expect the new path will go through R1, R5, R2 and R3. Traceroute NG confirms:

 

And the log file as well, showing you original path and the new path:

In this use case, we have leveraged several features of Traceroute NG. First, it runs continuously. Second, it detects, when a path is no longer available. Third, it can log results in a text format, that’s easy to share.

Now, enough boring reading, it’s time to try it out! You can download Traceroute NG here: https://www.solarwinds.com/free-tools/traceroute-ng/

 

We’re super excited to share this tool with the world and hope you find it useful.  Let us know your thoughts!

Filter Blog

By date: By tag:

SolarWinds uses cookies on its websites to make your online experience easier and better. By using our website, you consent to our use of cookies. For more information on cookies, see our cookie policy.