It took me a while to get the right combination of AIX SNMP Version 3 settings that will work with Authentication & Privacy enabled for Solar Winds.
Hopefully the steps below will help you get it working too. A list of sources which provided the clues I needed is at the end, its a long one as no single place had everything you will need. Hopefully this doco corrects that.
These steps worked ok on "AIX 6 TL08" and "AIX 7 TL 03".
- Install the snmp.crypto fileset to enable encryption (Obtained mine from AIX 6 Expansion Pack DVD 5765-G62 11/2012)
> lslpp -cl snmp.crypto
#Fileset:Level:PTF Id:State:Type:Description:EFIX Locked
/usr/lib/objrepos:snmp.crypto:188.8.131.52::COMMITTED:I:56-bit DES Encrypted SNMPV3 Support:
/etc/objrepos:snmp.crypto:184.108.40.206::COMMITTED:I:56-bit DES Encrypted SNMPV3 Support:
- Backup the /etc/rc.tcpip file as it's about to be modified
- Turn on encryption with the snmp switch command:
> /usr/sbin/snmpv3_ssw -e
This command will create symbolic links as required to enable/disable the encrypted/non-encrypted versions of snmpd & clsnmp
- Confirm and update /etc/rc.tcpip so that the following lines are no longer commented. Comment out dpid2 if it hasn't already been by the above.
# Start up the Simple Network Management Protocol (SNMP) daemon
start /usr/sbin/snmpd "$src_running"
# Start up the hostmibd daemon
start /usr/sbin/hostmibd "$src_running"
# Start up the snmpmibd daemon
start /usr/sbin/snmpmibd "$src_running"
# Start up the aixmibd daemon
start /usr/sbin/aixmibd "$src_running"
It looks like /usr/sbin/dpid2 functionality has been rolled into one of the above from at least AIX 6 TL08 onwards.
- Backup your existing snmp configuration files
- Pick one of your servers /etc/snmpd.boots files and make that file uniform across all your servers. Here's an example of its contents:
The first value is your EngineID, which can be something you made up, or one provided by the vendor. The second is the number of times snmp has been restarted. If you ensure this file is consistent across your AIX servers you can reuse your /etc/snmpdv3.conf file across them all. In turn, you can reuse the credentials when adding the nodes to Solarwinds. The auth/priv keys are married to the EngineID and won't work on another server if the EngineID is different there.
- Generate a new auth key with your local EngineID.
pwtokey -e -u auth <auth password> $(cat /etc/snmpd.boots | cut -f2 -d' ')
Display of 16 byte HMAC-MD5 privKey:
Display of 16 byte HMAC-MD5 localized privKey:
Make a note of the non-localized key value. E.g 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 of the two above. Also ensure you make a note of the passwords of course. You _must_ use the passwords when adding the node to Solarwinds. Attempts to use the key instead met with failure, likely due to the need for the EngineID to be paired up with the key somehow (context field didn't help).
- Generate a new priv key with your local EngineID. Again you're only interested in the non-localized key value. I'll use axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7 below for this.
pwtokey -e -u priv <priv password> $(cat /etc/snmpd.boots | cut -f2 -d' ')
- Clear your command history if you're worried about maintaining the privacy of these keys (good habit but a touch paranoid!)
- Update the /etc/clsnmp.conf file so it has an entry for local snmp testing. I'm using swro aka Solar-Winds-Read-Only. I might dabble with read-write later and want them segregated. Plug in your freshly generated auth/priv keys.
#winSnmpName targetAgent admin secName password context secLevel authProto authKey privProto privKey
swro 127.0.0.1 snmpv3 swro - - AuthPriv HMAC-MD5 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 DES axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7
You can replace the first "swro" with the local server name if you like. In fact multiple duplicate lines with each server/IP (and consistant snmpd.boots & snmpdv3.conf files) will allow you to kick off clsnmp commands from this server to any other that has been updated with this process. Handy if you want to setup scripting to pull specific MIB/OID values etc from all servers etc.
- Update your snmpd.peers file to ensure it has the details required for the snmpd process to access other components (e.g hostmibd/snmpmibd etc) for specific MIB/OID resources.
"gated" 220.127.116.11.18.104.22.168.22.214.171.124 "gated_password"
"dpid2" 126.96.36.199.188.8.131.52.184.108.40.206.1.2 "dpid_password"
"muxatmd" 220.127.116.11.18.104.22.168.22.214.171.124.1 "muxatmd_password"
# Enables cpu & volume information visibility to snmpd
"xmtopas" 126.96.36.199.188.8.131.52.184.108.40.206 "xmtopas_pw"
The default AIX set of "passwords" is being used above (and in the following snmpdv3.conf) which should get you sorted. Sing out if you spot any issues with this approach as it depends on locking out non-local access to snmpd via the snmpdv3.conf file except for auth/priv key holders.
For example, I didn't have an entry here for xmtopas. Once I put that in place the SolarWinds discovered resources list suddenly included "Volume Utilization" values aka filesystem and logical volume info. Once selected they appear in the "Asset Inventory" tab under logical volumes.
- Update your /etc/snmpdv3.conf file with the one below. Swap out the auth/priv keys with the ones you generated above.
## Solar Winds Specific Entries
USM_USER swro 00000002000000000A454172 HMAC-MD5 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 DES axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7 N -
VACM_GROUP swrogrp - swro readonly
VACM_VIEW swroview internet - included -
VACM_VIEW swroview 220.127.116.11.4.1.2 - included -
VACM_VIEW swroview 18.104.22.168.22.214.171.124 - included -
VACM_VIEW swroview 126.96.36.199.188.8.131.52 - included -
VACM_VIEW swroview 184.108.40.206.220.127.116.11 - included -
VACM_VIEW swroview 18.104.22.168.22.214.171.124 - included -
VACM_VIEW swroview directory - included -
VACM_VIEW swroview mgmt - included -
VACM_VIEW swroview mib-2 - included -
VACM_VIEW swroview system - included -
VACM_VIEW swroview aix - included -
VACM_VIEW swroview 126.96.36.199.4 - included -
VACM_VIEW swroview 188.8.131.52.6 - included -
VACM_VIEW swroview 184.108.40.206.220.127.116.11.5 - included -
VACM_VIEW swroview 18.104.22.168.4.1.2021 - included -
VACM_VIEW swroview 22.214.171.124.126.96.36.199.188.8.131.52.1.4 - included -
# Include snmpv3 managed MIBs with this view
VACM_VIEW swroview snmpModules - included -
# Include aixmibd managed MIBS with this view
VACM_VIEW swroview 184.108.40.206.220.127.116.11.191 - included -
VACM_ACCESS swrogrp - - AuthPriv - swroview - sworoview -
## AIX Internal SNMP Agent Specific Entries
# Allow localhost(only) SNMPv1 general access
COMMUNITY public public noAuthNoPriv 127.0.0.1 255.255.255.255 -
VACM_GROUP group1 SNMPv1 public -
VACM_ACCESS group1 - - noAuthNoPriv SNMPv1 defaultView - defaultView -
VACM_VIEW defaultView internet - included -
# Exclude snmpv3 related MIBs from the default view
VACM_VIEW defaultView snmpModules - excluded -
VACM_VIEW defaultView 18.104.22.168.22.214.171.124.4 - included -
VACM_VIEW defaultView 126.96.36.199.188.8.131.52.5 - included -
# Exclude aixmibd managed MIBS from this view
VACM_VIEW defaultView 184.108.40.206.220.127.116.11.191 - excluded -
# Access to data from gated/muxatmd/xmservd/dpid
smux 18.104.22.168.22.214.171.124.126.96.36.199 gated_password # gated
smux 188.8.131.52.184.108.40.206.220.127.116.11.1 muxatmd_password #muxatmd
smux 18.104.22.168.22.214.171.124.126.96.36.199 xmservd_pw #xmservd
smux 188.8.131.52.184.108.40.206.220.127.116.11.1.2 dpid_password #dpid
# These entries appear to be for IBM director at a guess
# They allow it to participate with the above
#VACM_GROUP director_group SNMPv2c public -
#VACM_ACCESS director_group - - noAuthNoPriv SNMPv2c defaultView - defaultView -
# Trap definitions
NOTIFY notify1 traptag trap -
TARGET_ADDRESS Target1 UDP 127.0.0.1 traptag trapparms1 - - -
TARGET_PARAMETERS trapparms1 SNMPv1 SNMPv1 public noAuthNoPriv -
## Global Defaults
# Set no access unless explicitly allowed by previous entries
DEFAULT_SECURITY no-access - -
# Set log location, maximum size, log level
logging file=/usr/tmp/snmpdv3.log enabled
#logging size=100000 level=0
logging size=100000 level=2
Still a work in progress locking down the AIX Internal SNMP agents and of course getting the right set of MIB included in the Solar Winds view. At least now I've something working I can fine tune and will be looking at other related posts here.
- Stop all snmp related services
stopsrc -s snmpmibd;stopsrc -s aixmibd;stopsrc -s snmpd;stopsrc -s hostmibd;stopsrc -s dpid2
- Start all snmp related servers (excluding the now redundant dpid2)
startsrc -s snmpmibd;startsrc -s aixmibd;startsrc -s snmpd;startsrc -s hostmibd
- Test things out locally by using the walk option on clsnmp (yup, IBM included a cleverly disguised snmpwalk command). I'm using the "internet" MIB in this example, lots of output!
clsnmp -h swro walk internet
If you get an error here, odds are you've a mismatched EngineID and auth/priv keys. Check out your /usr/tmp/snmpdv3.log for more details. As mentioned earlier the "swro" here is a reference to the matching line in /etc/clsnmp.conf. A server name could be used here (if defined there) instead and would result in a remote test.
- Once the dust settles, turn down the log level in snmpdv3.conf (level=0) to avoid excessive logging for daily operation.
Solar Winds Configuration
It should now be possible to add this node in Solarwinds.
- SNMP Version: SNMPv3
- SNMPv3 Username: swro
- SNMPv3 Authentication Method: MD5
- SNMPv3 Authentication Password: Use the password from pwtokey above (don't use the key, it doesn't work)
- SNMPv3 Privacy Method: DES56
- SNMPv3 Privacy Password: Use the password from pwtokey above (don't use the key, it doesn't work)
Hit the test button to see if all is ok. If it isn't make sure the services have been started and check the snmpdv3.log for more clues (especially the "did solar winds get here at all" clue).
As I've standardized the /etc/snmpd.boot and /etc/snmpdv3.conf files across the estate I can re-use these credentials and have saved them as "aix-swro".
Update 09/11/2015 - Post AIX upgrade (TL08/SP02 -> TL09/SP04) - Repair links to encrypted binaries
After the upgrade the links to the alternate, encrypted binaries for SNMP were reset to the default non-encrypted ones. This wasn't immediately apparent on Solarwinds until you try to "List Resources" for the node, which fails.
Solarwinds error: "<node> is currently down, unreachable, or provided credentials are not valid"
- Login as root on the target server
- Attempt an snmp walk to confirm this is the same problem
> clsnmp -h swro walk internet 1>/dev/null
Error reading file /etc/clsnmp.conf(Line 46): Invalid securityLevel
clsnmp: 1473-406 Error converting destinationName swro to Entity.
- Check the current snmp binaries being linked to
> ls -la /usr/sbin/snmpd /usr/sbin/clsnmp
lrwxrwxrwx 1 root system 9 Oct 14 12:02 /usr/sbin/snmpd -> snmpdv3ne
lrwxrwxrwx 1 root system 9 Oct 14 12:02 /usr/sbin/clsnmp -> clsnmpne
- Change the links to the encrypted snmpd binary with the command:
> snmpv3_ssw -e
In /etc/rc.tcpip file, comment out the line that contains: dpid2
In /etc/rc.tcpip file, remove the comment from the line that contains: snmpmibd
Stop daemon: snmpdMake the symbolic link from /usr/sbin/snmpd to /usr/sbin/snmpdv3e
Make the symbolic link from /usr/sbin/clsnmp to /usr/sbin/clsnmpe
Start daemon: snmpd
- Check the linked binary again
> ls -la /usr/sbin/snmpd /usr/sbin/clsnmpe
lrwxrwxrwx 1 root system 18 Nov 09 14:49 /usr/sbin/snmpd -> /usr/sbin/snmpdv3e
lrwxrwxrwx 1 root system 18 Nov 09 14:49 /usr/sbin/clsnmp -> /usr/sbin/clsnmpe
- On the solarwinds console, retry "list resources" on the node to confirm all is ok again
- IBM Doco
-- IBM SNMPv3 Documentation [http://www-01.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.networkcomm/snmpv3_intro.htm]
-- Creating users in SNMPv3 in AIX [http://www-01.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.networkcomm/HT_commadmn_create_snmpv3_user.htm#create_snmpv3_user]
-- AIX 7.1 snmpdv3.conf file setup [http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.files/snmpdv3.conf.htm?lang=en]
--- Useful to test an snmp configuration as this can interrogate your snmp server to confirm its working ok.
- Solar Winds Doco
-- Solarwinds SNMPV3 Implementation Guide [http://www.solarwinds.com/support/Orion/docs/Implementing_SNMPv3r1.pdf]
-- Managing SNMP credentials guidelines [http://www.solarwinds.com/netperfmon/solarwinds/wwhelp/wwhimpl/js/html/wwhelp.htm]
- Solar Winds Forums
-- Example SNMPv3 Configuration on AIX from Solarwinds forum 
-- Help with SNMPv3.1 on AIX [http://thwack.solarwinds.com/thread/42695]
-- Using SNMPv3.1 on AIX [http://thwack.solarwinds.com/thread/36507]
-- Configuring 3rd party Net-SNMP [http://thwack.solarwinds.com/thread/19323]
-- AIX Specific mods [http://thwack.solarwinds.com/community/application-and-server_tht/server-and-application-monitor/content?filterID=content~category[application-monitor-templates]&filterID=content~objecttype~objecttype[document]&query=unix]
- Make AIX a Solarwinds client on snmpv3 [http://odme.blogspot.com.au/2012/09/make-aix-solarwinds-client-on-snmpv3.html]
- Configuring AIX snmpd for MIB subagent access [http://odme.blogspot.com.au/2012/09/snmpdv3-wont-talk-to-mib-subagents.html]
-- The aixmbid, snmpmibd and similar subsystems depend on snmp to talk to each other! Here's the settings they use to do so.
-- Very detailed setup of the AIX snmpd for MIB subagent config including extra mib defaultviews [http://forums.cacti.net/viewtopic.php?t=19040]
- Nagios setup for snmp [http://nagios.frank4dd.com/howto/aix-snmp-setup.htm]
-- Examples on using snmpwalk
-- Example snmpdv3.conf for Nagios with highlights for nagios specifics
- Using snmpwalk [http://www.net-snmp.org/tutorial/tutorial-5/commands/snmpv3.html]