Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Alcatel 7550 Interface go from valid type to Uknown Status.

We have been running Soalrwinds for a few months across multiple Alcatel boxes and noticed that SolarWinds is sending Interfaces to an "uknown" status that we can see were known before. We can go back into the node, tell the system to manange the interfaces again and all looks ok. Over time we'll find uknown interfaces again which may or may not be the same ones.

Has anyone else seen this?

Thanks!

James

Find more posts tagged with

alcatel

unknown

Accepted answers

All comments

Yann

Hi,

Is the Alcatel device changing the Interface Index in the If-MIB ?

NPM is supposed to handle these changes but Alcatel devices might have created a new scenario which does not fall in the known ones...

If the issue occurs again, here is what you can do to troubleshoot it.

1) Extract the interfaces information for this nodes from the DB

SELECT * FROM Interfaces WHERE InterfaceID = X

You can find the interface ID in the URL in the web console when you are on an interface view.

2) Run a MIB Walk on the If-MIB - 1.3.6.1.2.1.2.2

See this post: Free SNMP MIB Browser

The idea is then to compare the results and see if there are discrepancies in the Interface indexes.

HTH,

Yann

jhilving

Thanks for the ideas Yann. I believe we did this before, but just to be sure I ran through one. I picked one of our routers which currently has 27 rows marked as "uknown". These interfaces are a combination of both physical and logical interfaces. Some of the logical interfaces have since been removed and some have not. The ones that have not all have the same ifindex as the Solarwinds DB.

Here is an example of running through one. I used the web to confirm the Index first.

I also checked the DB directly...just to be 100% we agree on frontend and backend database information.
--------------------------------

$sqsh -S SolarWindsDatabase -C "SELECT
[Interfaces].[Ifname],[Interfaces].[InterfaceIndex]
FROM
[Orion].[dbo].[Interfaces] Interfaces,
[Orion].[dbo].[Nodes] Nodes
WHERE
[Interfaces].[NodeID] = [Nodes].[NodeID]
AND Sysname = 'R1' AND [Interfaces].[Status] = '0'" -m vert

<snipped>
Ifname: 1/2/6
InterfaceIndex: 37945344

<snipped>
(27 rows affected)

--------------------------------

Next, I went to the router via SNMP and grep'd out the OIDs matching my index.
--------------------------------
$snmpwalk -On R1 .1.3.6.1.2.1.2.2 | grep 37945344
.1.3.6.1.2.1.2.2.1.1.37945344 = INTEGER: 37945344
.1.3.6.1.2.1.2.2.1.2.37945344 = STRING: 1/2/6, 10/100 Ethernet TX, "access_customer: "
.1.3.6.1.2.1.2.2.1.3.37945344 = INTEGER: ethernetCsmacd(6)
.1.3.6.1.2.1.2.2.1.4.37945344 = INTEGER: 9192
.1.3.6.1.2.1.2.2.1.5.37945344 = Gauge32: 100000000
.1.3.6.1.2.1.2.2.1.6.37945344 = STRING: 0:3:fa:d:b:79
.1.3.6.1.2.1.2.2.1.7.37945344 = INTEGER: up(1)
.1.3.6.1.2.1.2.2.1.8.37945344 = INTEGER: up(1)
.1.3.6.1.2.1.2.2.1.9.37945344 = Timeticks: (8380) 0:01:23.80
.1.3.6.1.2.1.2.2.1.10.37945344 = Counter32: 684155004
.1.3.6.1.2.1.2.2.1.11.37945344 = Counter32: 4541314
.1.3.6.1.2.1.2.2.1.12.37945344 = Counter32: 1307667
.1.3.6.1.2.1.2.2.1.13.37945344 = Counter32: 0
.1.3.6.1.2.1.2.2.1.14.37945344 = Counter32: 4033
.1.3.6.1.2.1.2.2.1.15.37945344 = Counter32: 0
.1.3.6.1.2.1.2.2.1.16.37945344 = Counter32: 2740178437
.1.3.6.1.2.1.2.2.1.17.37945344 = Counter32: 5210203
.1.3.6.1.2.1.2.2.1.18.37945344 = Counter32: 14978978
.1.3.6.1.2.1.2.2.1.19.37945344 = Counter32: 0
.1.3.6.1.2.1.2.2.1.20.37945344 = Counter32: 0
.1.3.6.1.2.1.2.2.1.21.37945344 = Gauge32: 0
.1.3.6.1.2.1.2.2.1.22.37945344 = OID: .0.0
--------------------------------

Thanks!

Yann

Thanks for the details.

Well next step would be to poll manually the device and run a wireshark trace and see if Orion gets results.

Open the web console. Admin > Manage Nodes. Select the node and click on More Actions > Poll Now.

It can take few seconds to send the packets as the job is added to the queue of the polling engine.

If you see results coming back to Orion, then let it poll by itself the device and compare both traces.

If you do not want to share the wireshark traces on this thread you can open a support ticket.

It is possible to enable Polling Debug as well, but be aware that the file can grow very quickly and slow your server, so you are warned . Ensure you disable the Polling Debug mode once done the tests.

We could see interesting things related to the alcatel devices.

Turn on poll debugging by creating this registry key then restart polling engine.

[HKEY_LOCAL_MACHINE\SOFTWARE\SolarWinds.Net\SWNetPerfMon\Settings]

"Debug Polling"="True"

Start -> Programs -> Solarwinds Orion -> Advanced Features -> Orion Service Manager. Restart the Solarwinds Network Performance Monitor

This will generate a file called polldebug.txt in the \Program Files\SolarWinds\Orion\ folder.

Once the debugging is complete, remove that registry key again, and restart the polling engine.

On 64 Bits OS:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\SolarWinds.Net\SWNetPerfMon\Settings

and the polldebug.txt file will be located in:

\Program Files (x86)\Solarwinds\Orion

HTH,

Yann

jhilving

Blahh! Seems that the problem is actually a tad different that the original report.

Before doing a debug, I took a peek at the DB one more time just to make sure I am seeing everything I expect.I was pretty confident Solarwinds was going to see the same results I am getting from my Unix CLI.

Now I am not the admin, and the previous admin has since left the company, so I cannot comment on what changes were made in the past to have caused this problem. I did, however, find something very interesting...two interface objects doing the same interface. I suspect this is really the root cause to the "uknown interface" mystery.
--------------------------------------
$sqsh -S SolarWindsDatabase -C "SELECT
*
FROM
    [Orion].[dbo].[Interfaces] Interfaces,
    [Orion].[dbo].[Nodes] Nodes
WHERE
[Interfaces].[NodeID] = [Nodes].[NodeID]
AND [Interfaces].[InterfaceIndex] = '37945344' AND SysName = 'R1'" \
-m vert \
| egrep -i "^nodeid:|^sysname:|^interfaceindex:|interfacename:|interfaceid:"

NodeID:                                83
InterfaceID:                           17494
InterfaceName:                         1/2/6, 10/100 Ethernet TX,
InterfaceIndex:                        37945344
SysName:                               R1

NodeID:                                83
InterfaceID:                           42816
InterfaceName:                         1/2/6, 10/100 Ethernet TX,
InterfaceIndex:                        37945344
SysName:                               R1
--------------------------------------

So we've been focusing on uknown status only, which shows why we might "think" the interfaces are coming in and out as the event summary is bouncing. Here I am showing a splitscreen of what the web is seeing and we can see the different results.

Now, the question has sort of changed to "What is the best way to correct the DB/Poller mess?". Should we go through and remove interface objects matching "uknown"? Should we remove entire nodes and rebuild? What do you think?

Does Solarwinds support some sort of API to control, or batch add, interface objects? My routers have hundreds of interfaces, this one in question has ~800, in the ifdesc table. I need to monitor only those interfaces that meet certain criterias....and those criterias are not as easy as "status=up" or "type=ethernet".

Thanks for the help!
J