Skip navigation

Geek Speak

14 Posts authored by: cyrussw

Orion Report Writer is a tool within Orion Network Performance Monitor (NPM) that allows users to format data and preview reports before displaying them. When running reports, sometimes it is not necessary or desirable to retrieve all the available information from a device. One question that has popped up is how to create a report that filters for certain days of the week. For example, rather than pulling a report that provides all information (24x7), to instead pull information that has been collected during normal business hours. This article concentrates on how to filter reports for specific times and days of the week. For more information on configuring reports in their entirety see Understanding Orion Report Writer.

 

The procedure for filtering is fairly simple and only involves making a few changes to the “Filter Results” section of the report. The following example shows a filter that returns information collected Monday thru Friday from 7 AM to 6 PM.


1.JPG


The report consist of two complex conditions. The first complex condition listed below consist of three statements grouped within an “all” condition. The first two filters within the “all” condition filter for the time of day. The filter spans from 7 AM to 6 PM. The 3rd statement filters for a device with the IP address of 10.110.68.115. Please note, rather than using an IP address in the filter, we can also customize it to use custom properties or other filtering techniques.


2.JPG


The second complex condition listed below consist of 5 statements grouped with an “any” condition. This condition states that if the report fall on any day of a normal work week (Monday thru Friday) to include it in the report. We can also convert the report to display weekends by changing the report to Saturday or Sunday.


3.JPG

Imagine that you are the systems administrator of your company and you are responsible for monitoring storage performance and capacity using SolarWinds Storage Manager. Suddenly there is a power outage at the company. Once power is restored, you find out that your database is either destroyed or corrupted. One way to avoid this issue is to perform a backup of your data. There are couple of quick ways to back up the Storage Manager database. Both will be explained below.

 

Performing a manual backup of the database:

 

1. Shutdown the mariadb service. This will also shut down all the other Storage Manager related services

 

2. Locate your database

 

  • Windows default location

 

  • Linux default location

 

3. Make a copy of the storage folder and place it in a safe location

 

4. Start the mariadb service and any other Storage Manager related services. The Storage Manager services are:

 

  • SolarWinds Storage Manager Collector
  • SolarWinds Storage Manager Event Receiver
  • SolarWinds Storage Manager Maintenance
  • SolarWinds Storage Manager Poller
  • SolarWinds Storage Manager Web Services

 

Running the DButil script:

 

Storage Manager ships with a script that will make backups of your database and even comes with the added feature of running maintenance routines to help keep the database running smoothly. This script can be automated to run regularly using Windows task scheduler in Microsoft Windows or crontab if you are running Linux. The user must make sure there is ample space in the destination folder he wishes to back up the Storage Manger database to.

 

Dbutil runs the following procedures:

 

  • Backup – Copies the database to a backup directory specified by the user
  • Maintenance – Runs database repair, analysis, and optimization procedures

 

Preparation to run dbutil in Windows:

  1. The default location of dbutil in Windows is %Program Files%\SolarWinds\Storage Manager Server\bin
  2. Using a text editor, edit your dbutil.bat file and find the following string: “Set BackupStage=C:\temp\mariadb_backup“
  3. Change the path to the directory where you would like to back up your Storage Manager database.

Note: It is recommended that you do not use a directory within the Storage Manager install directory.

Preparation to run dbutil in Linux:

  1. The default location of dbutil in Linux is /opt/Storage_Manager_Server/bin
  2. Using a text editor, edit your dbutil.sh file and find the following string: export backupStage=/opt/mariadb_backup
  3. Change the path /opt/mariadb_backup to the directory where you would like to back up your Storage Manager database.

Note: It is recommended that you do not use a directory within the Storage Manager install directory.

Usage of dbutil:

  • Windows - dbutil.bat [backup|maintenance]
  • Linux -  dbutil.sh [backup|maintenance]

More information can be found here for running dbutil.

The Storage Manager Collector service is responsible for collecting data from proxy agents and transferring any collected information to the database. When customers add additional devices to Storage Manager, this can cause additional database connection request to the database. It is possible that the collector will not be able to store information in the database due to too many connections. I will discuss how to get around this issue.

 

Note: Storage Manager versions 5.6 and newer use MariaDB. For previous versions, MySQL is used. For versions prior to 5.6, substitute MySQL for MariaDB in the following instructions.

 

First we need to check the collector log file to see what messages are being generated. The collector log file can be found in the following location:

 

  • Windows - %Program Files%\SolarWinds\Storage Manager Server\webapps\ROOT\logs

 

  • Linux - /opt/Storage_Manager_Server/webapps/ROOT/logs

 

The file name is called mod.adm.collect.Collector.log.

 

When viewing this file look for the following error:

 

[Collect list digestor] Non-recoverable database error [08004]:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections"

 

The error tells us there are too many connections trying to hit the database at once.

 

To fix this issue requires modifying the database configuration file called my.cnf. This file can be found in the following location:

 

  • Windows - %Program Files%\SolarWinds\Storage Manager Server\mariadb

 

  • Linux - /opt/Storage_Manager_Server/mariadb

 

Open the my.cnf file with a text editor and search for the entry max_connections=. The default value is 400. Change it to 600, save the file and restart the Storage Manager services including the database service.

 

Note: Do not exceed 600 when changing this value.

Sometimes when installing an application on a server, users will run into the issue where the desired port is being blocked by another application. Rather than installing 3rd party software to determine what application is blocking the port, we can use tools that already come standard with Windows. Here we will discuss one way of determining what application is using what port.

 

There are two tools we will need:

  • Command prompt
  • Windows Task manager

 

For example let’s assume that a syslog application was installed on a server only to find out that port 514 is already in use. To determine what application is blocking the port we must perform the following steps:

 

      1. First we notice that port 514 is already in use:

Capture1.JPG

 

 

      2. Bring up a command prompt and run the command netstat –a –o –p udp | find “514”

 

Notes:

    • -a displays all connections and listening ports
    • -o displays the process id (very important)
    • -p displays the desired protocol (here we are interested in UDP)
    • We can also specify the specific port of interest. Here we are interested in port 514


Capture2.JPG

 

     3. Note that the Process ID (PID) associated with the blocked port is 6724. Now we need to find out what application owns PID 6724. To do this we bring up the Windows Task Manager and under “View” and     

        “Select Columns…” select the option for “PID (Process Identifier).”


Capture3.JPG

 

 

     Next Click “Ok” and you will now see the PIDs displayed in Windows Task Manager.

 

     4. When I search for the application associated with PID 6724 we can see that SyslogService.exe*32 is using port 514.


Capture4.JPG

 

     Our options are to either use the syslog server currently running or kill the PID which will make port 514 available.

By default Storage Manager places the database and it install files on the same drive. Over time the database will expand. It is important to verify there is sufficient disk space before performing an upgrade of Storage Manager. We must have twice the amount of free space as the largest table in the database. During the upgrade, Storage Manager will build temporary database tables from the actual database and it is because of this that we must verify sufficient disk space on the drive.

 

MariaDB creates a number of different data files in the mariadb directory. The file types include:

  • .frm – format (schema) file
  • .MYD – data file
  • .MYI – index file

 

It is the *.MYD (data) file that we must check. Within the database directory we must sort the files from largest to smallest keeping track of the largest .MYD file in that directory.

 

The database can be found at the following location:

 

  • Windows - <installed drive>\Program Files\SolarWinds\Storage Manager Server\mariadb\data\storage  directory

 

  • Linux - <installed path>/Storage_Manager_Server/mariadb/data/storage directory

 

Notes:

 

  • This is the default location of where the temporary database tables are created.

 

  • Storage Manager versions 5.6 and newer use MariaDB. For previous versions, MySQL is used. For versions prior to 5.6, substitute MySQL for MariaDB

 

If you have insufficient disk space, you must point the temporary database tables to a drive that have sufficient space before upgrading. For more information on how to relocate the temporary database tables please see the Storage Manager Administrator Guide.

cyrussw

EOC and HTTPS

Posted by cyrussw Dec 6, 2013

Security is always a good thing when it comes to websites and confidential information. When a website uses Hyper Text Transfer Protocol (HTTP), it can be more vulnerable to attacks vs. using Hyper Text Transfer Protocol Security (HTTPS) which is HTTP integrated with the Secure Socket Layer (SSL). It is the SSL which encrypts the HTTP data before sending it to the destination. One of the main advantages of HTTPS is that the information is sent encrypted from the source to the destination. HTTP does not have this functionality. The SolarWinds EOC website allows users to utilize HTTPS thus securing transmissions from the webserver to the end user.

 

The following explains how to configure HTTPS on Windows Server 2003, 2008, and 2012 for the EOC website:

 

To enable SSL Connections to the SolarWinds EOC web site in Windows Server 2003:

  1. Log on as an administrator to your SolarWinds EOC server.
  2. Click Start > Control Panel > Administrative Tools > Computer Management.
  3. Expand Services and Applications > Internet Information Services (IIS) Manager > Web Sites.
  4. Click SolarWinds EOC and then click Action > Properties.
  5. Click the Web Site tab.
  6. Confirm that SSL port is set to 443.
  7. Click Advanced.
  8. If the Multiple SSL identities for this Web site field does not list the IP address for the SolarWinds Web Console with SSL port 443, complete the following steps:

a.   Click Add and then select the IP address of the SolarWinds Web Console.

    • As it was set initially in the Configuration Wizard, this option is usually set to (All Unassigned). If the IP address of the SolarWinds Web Console was not initially set to (All Unassigned), select the actual, configured IP address of the SolarWinds Web Console.

b.   Type 443 as the TCP port, and then click OK.

   9. If you want to accept only HTTPS connections, complete the following steps:

    1. Click the Directory Security tab.
    2. Click Edit in the Secure communications section.
    3. Select Require secure channel (SSL).
    4. Select Accept client certificates in the Client certificates area.
    5. Click OK on the Secure Communications window.

  10. Click Apply and then click OK to exit.

 

To enable SSL Connections to the SolarWinds EOC web site in Windows Server 2008:

  1. Log on as an administrator to your SolarWinds EOC server.
  2. Click Start > Administrative Tools > Internet Information Services (IIS) Manager.
  3. In the Connections pane, expand the name of your SolarWinds EOC server, and then expand Sites.
  4. Select your SolarWinds EOC web site, and then click Bindings in the Actions pane on the right.
  5. Click Add in the Site Bindings window.
  6. In the Type field, select https, and then confirm that the Port value is 443.
  7. In the SSL Certificate field, select a certificate, and then click OK.
  8. Click Close on the Site Bindings window.
  9. In the center pane, double-click SSL Settings in the IIS group.
  10. Select Require SSL, and then click Apply in the Actions pane on the right.
  11. In the Connections pane, select your SolarWinds EOC web site.
  12. Click Restart in the Manage Web Site group on the right.

 

To enable SSL Connections to the SolarWinds EOC web site in Windows Server 2012:

  1. Log on as an administrator to your SolarWinds EOC server.
  2. Click Settings > Control Panel > Administrative Tools > Internet Information Services (IIS) Manager.
  3. In the Connections pane, expand the name of your SolarWinds EOC server, and then expand Sites.
  4. Select your SolarWinds EOC web site, and then click Bindings in the Actions pane on the right.
  5. Click Add in the Site Bindings window.
  6. In the Type field, select https, and then confirm that the Port value is 443.
  7. In the SSL Certificate field, select a certificate, and then click OK.
  8. Click Close on the Site Bindings window.
  9. In the center pane, double-click SSL Settings in the IIS group.
  10. Select Require SSL, and then click Apply in the Actions pane on the right.
  11. In the Connections pane, select your SolarWinds EOC web site.
  12. Click Restart in the Manage Web Site group on the right.

Normally it is good to just have the Storage Manager Agent or Proxy Agent running on a server as the only application. Unfortunately this cannot always be the case. There are situations where a user will run multiple applications on the same server with the Agent. At times there will also be multiple NIC cards on the server to cater to the different applications which may be routing to different subnets. In this type or scenario the Agent will use the first available NIC card on the server thus meaning it could cause traps to route to the wrong subnet causing the Storage Manager Server to loose visibility to the Agent. If the Agent is installed on a server with 2 or more installed network cards, then the core.xml file on the Agent must be manually updated to define the IP address for the network card you want associated with sending messages to the Storage Manager Server. For example, if eth0 and eth1 are installed on your server, and eth1 is used to send traps, you must associate the eth1 IP address to the Agent by following these instructions. By doing this you will allow the Agent to route traps to the Storage Manager Server.


For Windows servers complete the following steps:

  1. Navigate to \Storage Manager Agent\. Example: C:\Program Files\SolarWinds\Storage Manager Agent\
  2. Open core.xml in a text editor.
  3. Enter the IP address of the network card you want associated with sending traps.
  4. Save the file.
  5. Restart the Storage Manager Agent Service.

 

  For Linux Servers, complete the following steps: 

  1. Open /etc/hosts in a text editor.
  2. Keep the default options, and add the IP address, domain, and hostnames for the network card you want associated with sending traps.
  3. Save the file.
  4. Make sure the core.xml file includes the IP address in the /etc/hosts file.
  5. Restart the Storage Manager Agent Services.

Most people use some type of anti-virus software on their servers to protect against unwanted invasions. Having anti-virus security is good yet it can also come with a caveat when running in parallel with Storage Manager software. The problem we could encounter is file locks caused by real time scans and other security options such as blocked ports. This can cause problems with Storage Manager and monitoring. To get around this issue it is recommended to configure anti-virus software to avoid scans of the Storage Manager software and database.

 

It is recommended to add exclusions to your anti-virus and intrusion detection software when it is running on the same sever as Storage Manager.

 

Exclusions:

 

Storage Manager Server for Windows Exclusions:

 

  • Install Directory\Storage Manager Server\agent\systemic
  • Install Directory\Storage Manager Server\agent\administrative
  • Install Directory\Storage Manager Server\mariadb

                 Note: Storage Manager versions 5.6 and newer use MariaDB. For previous versions, MySQL is used. For versions prior to 5.6, substitute MySQL for MariaDB

  • C:\Windows\Temp

 

Storage Manager Agent for Linux Exclusion:

 

  • Install Directory/Storage_Manager_Server/agent/systemic
  • InstallDirectory/Storage_Manager_Server/agent/administrative
  • Install Directory/Storage_Manager_Server/mariadb

 

Storage Manager Agent for Windows Exclusion:

 

  • Install Directory\Storage Manager Agent\systemic
  • Install Directory\Storage Manager Agent\administrative

 

Storage Manager Agent for Non Windows Exclusion:

 

  • Install Directory\Storage_Manager _Agent/systemic
  • Install Directory\Storage_Manager _Agent/administrative

 

Port Exclusions:

 

Your anti-virus software must allow access to the following ports:

 

  • TCP 4319
  • UDP 162, 10162, 20162

Troubleshooting polling issues can involve many steps and there are a few quick things we can try first before placing a support call. For example, say we have a piece of networking equipment such as a router or switch that is being monitored in Orion and suddenly the interfaces go unknown (gray) on the network device. This is telling us that SNMP information is not being recorded for the interfaces. More than likely we will not see any CPU or Memory utilization being reported either. To troubleshoot this issue we can do the following.

  1. Verify that the network device is configured correctly for sending SNMP traffic to the Network Management Software. A quick example would be verifying that the community string is correct.
  2. Run a packet capture on the server hosting the Network Management Software. We can poll the device from the Orion website by selecting “poll now” from Node Details. This sends SNMP (UDP 161) traps to the destination network device. While you are polling the device, run a sniffer trace using a program like or similar to Wireshark (formerly Ethereal). If there is a network issue, you will see SNMP frames leaving your Network Management Server, while no response is returned from the network device.
  3. If you do see frames returning from the network device, then this rules out network connectivity issues. Next verify there is no firewall blocking the SNMP frames on the server hosting the Network Management Software. If you are running a sniffer such as Wireshark, the incoming frames are being read at the Network Interface Card. A local server firewall such as Windows Firewall can still block SNMP frames from reaching the Network Management Software even though we see them hitting the NIC card. Note also that if a firewall is blocking SNMP traffic on the local server this will prevent any SNMP related information from being reported in your Network Management Software.

In summary verifying end to end connectivity between the Network Management Server and the device in question is the key and there are a few steps we can do to verify and possibly fix polling issues saving the end user some time.

By default the Storage Manager Website uses port 9000. Some customer may wish to change this to another port due to port availability or firewall issues. There is a configuration file within the install subdirectory of Storage Manager that will allow users to change from default port 9000 to another port.

 

In this example we will change the Storage Manager Website port to 80. This example will also show you how to change the port assignment for Storage Manager running on Windows or Linux.

The file we must modify is called server.xml and it can be found in the following locations:

 

  • Windows - <installed drive>:\Program Files\SolarWinds\Storage Manager Server\conf
  • Linux - /opt/Storage_Manager_Server/conf


Open the file with a text editor and do a search for “9000.” You will see a section that looks like the following snippet:

 

<Connector port="9000" protocol="HTTP/1.1" URIEncoding="UTF-8"

               disableUploadTimeout="true" connectionTimeout="20000"

acceptCount="100" redirectPort="8443" enableLookups="false"

maxSpareThreads="75" minSpareThreads="25"

maxThreads="150" maxHttpHeaderSize="8192"/>


The entry for Connector port= is where we change the port assignment for the Storage Manager Website. I change the entry from 9000 to 80.

 

<Connector port="80" protocol="HTTP/1.1" URIEncoding="UTF-8"

disableUploadTimeout="true" connectionTimeout="20000"

acceptCount="100" redirectPort="8443" enableLookups="false"

maxSpareThreads="75" minSpareThreads="25"

maxThreads="150" maxHttpHeaderSize="8192"/>


Save your changes to the file. Finally we must restart the Storage Manager Web service before the change will take effect. To restart the Storage Manager Web service do the following:

 

  • Windows - Run services.msc and restart the SolarWinds Storage Manager Web Services service
  • Linux – From the Command Line Interface type “/etc/init.d/storage_manager_server restart webserver”


Open the Storage Manager website in a web browser referencing your new port assignment.

Issue:

Sometimes files are left behind when uninstalling Storage Manager in Linux, and when re installing on the same server a message is received saying it is already installed.

 

Resolution:

    • Check the /usr/bin sub directory and verify there are no Storage Manager scripts present. If so, delete them.
    • If you are installing Storage Manager Server, make sure the opts/Storage_Manager_Server directory is gone. If you are installing the Storage Manager Agent software, make sure the opts/Storage_Manager_Agent directory is gone. If it is present, delete it.
    • Go to etc/init.d and make sure storage_manager_server file is not there if you are installing Server. For Storage Manager Agent make sure storage_manager_agent file is not present. If so, delete it.
    • Proceed with reinstalling Storage Manager Agent or Server software.

Issue:

 

When installing EOC it will also install Orion modules.

eoc.png

 

When you uninstall EOC via "add/remove programs" or  "programs and features", the uninstall process will only uninstall "SolarWinds Enterprise Operations Console." The modules will remain. When you install EOC to a different drive and run the configuration wizard you will receive the error:

 

"error while executing script- column name or number of supplied values does not match table definition"


Resolution:




Make sure EOC and all of it's associated modules are uninstalled before moving EOC to a different drive on the same system.

Storage Manager allows users to access the Storage Manager Database via the Storage Manager Website. This feature can be useful when you need to run queries against the database.

 

Note: It is recommended that you perform a full backup of your database before running any manual queries.

 

Accessing the database simply involves changing the Storage Manager URL. We have two options for accessing the database

 

- RunScriptServlet - Allow us to run queries to the database

 

- DatabaseAdmin - Allow us to run queries to the database and also give other useful information about the Storage Manager database

 

In the screen shot below, the URL for the Storage Manager website is "http://10.110.68.112:9000/LoginServlet."

 

Capture1.PNG


To open a window to the Storage Manager Database we would change the URL to the following:


http://10.110.68.112:9000/RunScriptServlet

or

http://10.110.68.112:9000/DatabaseAdmin

See screen shots below.

 

Capture2.PNG

 

Capture3.PNG

Storage manager uses Tomcat for the web server. The session timeout parameter can be found in the web.xml file.

 

For windows the path will be:

 

<installed drive>\Program Files\SolarWinds\Storage Manager Server\webapps\ROOT\WEB-INF

 

Linux:

 

/Storage_Manager_Server/webapps/ROOT/WEB-INF

 

Within the WEB-INF subdirectory there will be a file called web.xml. Open this file with a text editor and do a search for <session-timeout>.

 

The default will be set for 30 minutes. If we want to change the timeout to 1 hour, we simply change the value to 60.

 

Before:

 

<session-timeout>30</session-timeout>

 

After:

 

<session-timeout>60</session-timeout>

 

 

If you wish to set the timeout to infinity, change the value to -1.

 

 

Once the changes have been made save the file and restart the Storage Manager Web Service.

 

 

To restart the service for Windows, run services.msc, next locate the SolarWinds Storage Manager Web Service and select restart.

 

 

1.jpg

 

 

To restart the web service in Linux, open a SSH session to the Storage Manager Server and log in with an account that has proper permissions such as root and type the following command:

 

 

/etc/init.d/storage_manager_server restart webserver

 

2.jpg

 

Note that upgrading or performing an uninstall and re install of Storage Manager will set the timeout value back to the default of 30 minutes.

Filter Blog

By date: By tag: