Okay everyone so here is my issue. I work for a large ISP out of Minnesota and we just stood up a brand new all Cisco network consisting entirely of ASR 9K routers, specifically 9001's and 9010's. When first adding them to Orion before we upgraded to 11.5 we ran into a lot of hardware health alarms, mainly relating to transmit and receive power levels. When heard about the new feature in 11.5 and that we should have the ability to disable and fine tune thresholds on individual alarms. After we upgraded to 11.5 and started to add the bulk of our new routers into NPM and NCM I noted the alarms that were triggered and attempted to disable the individual alarms.
At first it looked like the alarms were starting to clear, it did take quite a while for some of them but I thought that may be the case. Then I started to run into a few devices that the sensors just would not disable. I would disable them and exit out then go back in right away and check and it would say disabled, but after a fair amount of time went by, long enough that it should have cleared, I would check again and it would say enabled again. Also about 80% of the alarms that had cleared after I disabled them all or a sudden popped back into alarm and all the sensors were re-enabled.
I tried just about everything I could think of and was unable to find any posts even slightly related to my issue at that time. I do see a few threads now with issues that are kind of similar, but they all seem to be relating to servers, but my issue is with major core transport routers. I did manage to find one way to get the alarms to clear that worked a majority of the time, but is in now way a fix for the issue. if I went in an edited the threshold and forced the sensor to up they would normally clear, but now they are no longer visible like they would be if they were disabled.
Has anyone experienced any similar issues with IP Transport related equipment or with routers of the same or similar models? If you have any ideas please share because I'm open for just about anything at this point. I opened a support case regarding the issue, but since they said they are pretty backed up I figured I would throw this out there and see if I got any ideas before I got a response from them.