Configuring SNMP Version 3 on AIX

Version 3

    It took me a while to get the right combination of AIX SNMP Version 3 settings that will work with Authentication & Privacy enabled for Solar Winds.

    Hopefully the steps below will help you get it working too. A list of sources which provided the clues I needed is at the end, its a long one as no single place had everything you will need. Hopefully this doco corrects that.

     

    AIX Configuration

    These steps worked ok on "AIX 6 TL08" and "AIX 7 TL 03".

     

    • Install the snmp.crypto fileset to enable encryption (Obtained mine from AIX 6 Expansion Pack DVD 5765-G62 11/2012)

    > lslpp -cl snmp.crypto

    #Fileset:Level:PTF Id:State:Type:Description:EFIX Locked

    /usr/lib/objrepos:snmp.crypto:6.1.2.0::COMMITTED:I:56-bit DES Encrypted SNMPV3 Support:

    /etc/objrepos:snmp.crypto:6.1.2.0::COMMITTED:I:56-bit DES Encrypted SNMPV3 Support:

     

    • Backup the /etc/rc.tcpip file as it's about to be modified
    • Turn on encryption with the snmp switch command:

    > /usr/sbin/snmpv3_ssw -e

    This command will create symbolic links as required to enable/disable the encrypted/non-encrypted versions of snmpd & clsnmp

    • Confirm and update /etc/rc.tcpip so that the following lines are no longer commented. Comment out dpid2 if it hasn't already been by the above.

    ---cut---

    # Start up the Simple Network Management Protocol (SNMP) daemon

    start /usr/sbin/snmpd "$src_running"

     

    # Start up the hostmibd daemon

    start /usr/sbin/hostmibd "$src_running"

     

    # Start up the snmpmibd daemon

    start /usr/sbin/snmpmibd "$src_running"

     

    # Start up the aixmibd daemon

    start /usr/sbin/aixmibd "$src_running"

    ---cut---

    It looks like /usr/sbin/dpid2 functionality has been rolled into one of the above from at least AIX 6 TL08 onwards.

    • Backup your existing snmp configuration files

    ---cut---

    /etc/snmpdv3.conf

    /etc/clsnmp.conf

    /etc/snmpd.boots

    /etc/snmpd.peers

    ---cut---

    • Pick one of your servers /etc/snmpd.boots files and make that file uniform across all your servers. Here's an example of its contents:

    ---cut---

    00000002000000000A454172 0000000082

    ---cut---

    The first value is your EngineID, which can be something you made up, or one provided by the vendor. The second is the number of times snmp has been restarted. If you ensure this file is consistent across your AIX servers you can reuse your /etc/snmpdv3.conf file across them all. In turn, you can reuse the credentials when adding the nodes to Solarwinds. The auth/priv keys are married to the EngineID and won't work on another server if the EngineID is different there.

    • Generate a new auth key with your local EngineID.

    ---cut---

    pwtokey -e -u auth <auth password> $(cat /etc/snmpd.boots | cut -f2 -d' ')

     

    Display of 16 byte HMAC-MD5 privKey:

    5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0

     

    Display of 16 byte HMAC-MD5 localized privKey:

    axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx5

    ---cut---

    Make a note of the non-localized key value. E.g 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 of the two above. Also ensure you make a note of the passwords of course. You _must_ use the passwords when adding the node to Solarwinds. Attempts to use the key instead met with failure, likely due to the need for the EngineID to be paired up with the key somehow (context field didn't help).

    • Generate a new priv key with your local EngineID. Again you're only interested in the non-localized key value. I'll use axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7 below for this.

    ---cut---

    pwtokey -e -u priv <priv password> $(cat /etc/snmpd.boots | cut -f2 -d' ')

    ---cut---

    • Clear your command history if you're worried about maintaining the privacy of these keys (good habit but a touch paranoid!)

    ---cut---

    > ~/.sh_history

    ---cut---

    • Update the /etc/clsnmp.conf file so it has an entry for local snmp testing. I'm using swro aka Solar-Winds-Read-Only. I might dabble with read-write later and want them segregated. Plug in your freshly generated auth/priv keys.

    ---cut---

    /etc/clsnmp.conf

     

    #winSnmpName  targetAgent  admin  secName  password  context secLevel authProto  authKey                           privProto  privKey

    #----------------------------------------------------------------------------------------------------------------------------------------------------------

    swro          127.0.0.1    snmpv3 swro     -         -       AuthPriv HMAC-MD5   5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0  DES axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7

    ---cut---

    You can replace the first "swro" with the local server name if you like. In fact multiple duplicate lines with each server/IP (and consistant snmpd.boots & snmpdv3.conf files) will allow you to kick off clsnmp commands from this server to any other that has been updated with this process. Handy if you want to setup scripting to pull specific MIB/OID values etc from all servers etc.

    • Update your snmpd.peers file to ensure it has the details required for the snmpd process to access other components (e.g hostmibd/snmpmibd etc) for specific MIB/OID resources.

    ---cut---

    /etc/snmpd.peers

     

    ###############################################################################

    #

    "gated"     1.3.6.1.4.1.2.3.1.2.1.2     "gated_password"

    "dpid2"     1.3.6.1.4.1.2.3.1.2.2.1.1.2 "dpid_password"

    "muxatmd" 1.3.6.1.4.1.2.3.1.2.3.1.1 "muxatmd_password"

    #

    # Enables cpu & volume information visibility to snmpd

    "xmtopas"       1.3.6.1.4.1.2.3.1.2.1.3 "xmtopas_pw"

    #

    ## EOF

    ---cut---

    The default AIX set of "passwords" is being used above (and in the following snmpdv3.conf) which should get you sorted. Sing out if you spot any issues with this approach as it depends on locking out non-local access to snmpd via the snmpdv3.conf file except for auth/priv key holders.

    For example, I didn't have an entry here for xmtopas. Once I put that in place the SolarWinds discovered resources list suddenly included "Volume Utilization" values aka filesystem and logical volume info. Once selected they appear in the "Asset Inventory" tab under logical volumes.

    • Update your /etc/snmpdv3.conf file with the one below. Swap out the auth/priv keys with the ones you generated above.

    ---cut---

    ##

    ## Solar Winds Specific Entries

    ##

    #

    USM_USER swro 00000002000000000A454172 HMAC-MD5 5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 DES axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7 N -

    #

    VACM_GROUP swrogrp - swro readonly

    #

    VACM_VIEW swroview internet                    - included -

    VACM_VIEW swroview 1.3.6.1.4.1.2               - included -

    VACM_VIEW swroview 1.3.6.1.4.1.2.2             - included -

    VACM_VIEW swroview 1.3.6.1.4.1.2.3             - included -

    VACM_VIEW swroview 1.3.6.1.4.1.2.5             - included -

    VACM_VIEW swroview 1.3.6.1.4.1.2.6             - included -

    VACM_VIEW swroview directory                   - included -

    VACM_VIEW swroview mgmt                        - included -

    VACM_VIEW swroview mib-2                       - included -

    VACM_VIEW swroview system                      - included -

    VACM_VIEW swroview aix                         - included -

    VACM_VIEW swroview 1.3.6.1.4                   - included -

    VACM_VIEW swroview 1.3.6.1.6                   - included -

    VACM_VIEW swroview 1.3.6.1.6.3.1.1.5           - included -

    VACM_VIEW swroview 1.3.6.1.4.1.2021            - included -

    VACM_VIEW swroview 1.3.6.1.4.1.2.3.1.2.2.2.1.4 - included -

    #

    # Include snmpv3 managed MIBs with this view

    VACM_VIEW swroview snmpModules                 - included -

    # Include aixmibd managed MIBS with this view

    VACM_VIEW swroview 1.3.6.1.4.1.2.6.191         - included -

    #

    VACM_ACCESS swrogrp - - AuthPriv - swroview - sworoview -

     

     

    ##

    ## AIX Internal SNMP Agent Specific Entries

    ##

    #

    # Allow localhost(only) SNMPv1 general access

    COMMUNITY public public noAuthNoPriv 127.0.0.1 255.255.255.255  -

    VACM_GROUP group1 SNMPv1  public  -

    VACM_ACCESS  group1 - - noAuthNoPriv SNMPv1   defaultView - defaultView -

    #

    VACM_VIEW defaultView internet                    - included -

    # Exclude snmpv3 related MIBs from the default view

    VACM_VIEW defaultView snmpModules                 - excluded -

    VACM_VIEW defaultView 1.3.6.1.6.3.1.1.4           - included -

    VACM_VIEW defaultView 1.3.6.1.6.3.1.1.5           - included -

    # Exclude aixmibd managed MIBS from this view

    VACM_VIEW defaultView 1.3.6.1.4.1.2.6.191         - excluded -

    #

    # Access to data from gated/muxatmd/xmservd/dpid

    smux   1.3.6.1.4.1.2.3.1.2.1.2   gated_password  # gated

    smux   1.3.6.1.4.1.2.3.1.2.3.1.1   muxatmd_password #muxatmd

    smux   1.3.6.1.4.1.2.3.1.2.1.3   xmservd_pw   #xmservd

    smux   1.3.6.1.4.1.2.3.1.2.2.1.1.2   dpid_password   #dpid

    #

    #

    # These entries appear to be for IBM director at a guess

    #  They allow it to participate with the above

    #VACM_GROUP director_group SNMPv2c public -

    #VACM_ACCESS director_group - - noAuthNoPriv SNMPv2c defaultView - defaultView -

    #

    # Trap definitions

    NOTIFY notify1 traptag trap -

    TARGET_ADDRESS Target1 UDP 127.0.0.1 traptag trapparms1 - - -

    TARGET_PARAMETERS trapparms1 SNMPv1  SNMPv1  public  noAuthNoPriv -

     

     

    ##

    ## Global Defaults

    ##

    #

    # Set no access unless explicitly allowed by previous entries

    DEFAULT_SECURITY no-access - -

    #

    # Set log location, maximum size, log level

    logging         file=/usr/tmp/snmpdv3.log       enabled

    #logging         size=100000                     level=0

    logging         size=100000                     level=2

    #

    ## EOF

    ---cut---

    Still a work in progress locking down the AIX Internal SNMP agents and of course getting the right set of MIB included in the Solar Winds view. At least now I've something working I can fine tune and will be looking at other related posts here.

    • Stop all snmp related services

    ---cut---

    stopsrc -s snmpmibd;stopsrc -s aixmibd;stopsrc -s snmpd;stopsrc -s hostmibd;stopsrc -s dpid2

    ---cut---

    • Start all snmp related servers (excluding the now redundant dpid2)

    ---cut---

    startsrc -s snmpmibd;startsrc -s aixmibd;startsrc -s snmpd;startsrc -s hostmibd

    ---cut---

    • Test things out locally by using the walk option on clsnmp (yup, IBM included a cleverly disguised snmpwalk command). I'm using the "internet" MIB in this example, lots of output!

    ---cut---

    clsnmp -h swro walk internet

    ---cut---

    If you get an error here, odds are you've a mismatched EngineID and auth/priv keys. Check out your /usr/tmp/snmpdv3.log for more details. As mentioned earlier the "swro" here is a reference to the matching line in /etc/clsnmp.conf. A server name could be used here (if defined there) instead and would result in a remote test.

    • Once the dust settles, turn down the log level in snmpdv3.conf (level=0) to avoid excessive logging for daily operation.

     

    Solar Winds Configuration

     

    It should now be possible to add this node in Solarwinds.

    Key Values

    • SNMP Version: SNMPv3
    • SNMPv3 Username: swro
    • SNMPv3 Authentication Method: MD5
    • SNMPv3 Authentication Password: Use the password from pwtokey above (don't use the key, it doesn't work)
    • SNMPv3 Privacy Method: DES56
    • SNMPv3 Privacy Password: Use the password from pwtokey above (don't use the key, it doesn't work)

     

    Hit the test button to see if all is ok. If it isn't make sure the services have been started and check the snmpdv3.log for more clues (especially the "did solar winds get here at all" clue).

     

    As I've standardized the /etc/snmpd.boot and /etc/snmpdv3.conf files across the estate I can re-use these credentials and have saved them as "aix-swro".

    swSNMP01.png

     

     

    Update 09/11/2015 - Post AIX upgrade (TL08/SP02 -> TL09/SP04) - Repair links to encrypted binaries

     

    After the upgrade the links to the alternate, encrypted binaries for SNMP were reset to the default non-encrypted ones. This wasn't immediately apparent on Solarwinds until you try to "List Resources" for the node, which fails.

    Solarwinds error: "<node> is currently down, unreachable, or provided credentials are not valid"

     

    • Login as root on the target server
    • Attempt an snmp walk to confirm this is the same problem

    > clsnmp -h swro walk internet 1>/dev/null

    Error reading file /etc/clsnmp.conf(Line 46):    Invalid securityLevel

    clsnmp: 1473-406 Error converting destinationName swro to Entity.

    • Check the current snmp binaries being linked to

    > ls -la /usr/sbin/snmpd /usr/sbin/clsnmp

    lrwxrwxrwx 1 root system 9 Oct 14 12:02 /usr/sbin/snmpd -> snmpdv3ne

    lrwxrwxrwx 1 root system 9 Oct 14 12:02 /usr/sbin/clsnmp -> clsnmpne

    • Change the links to the encrypted snmpd binary with the command:

    > snmpv3_ssw -e

    In /etc/rc.tcpip file, comment out the line that contains: dpid2

    In /etc/rc.tcpip file, remove the comment from the line that contains: snmpmibd

    Stop daemon: snmpdMake the symbolic link from /usr/sbin/snmpd to /usr/sbin/snmpdv3e

    Make the symbolic link from /usr/sbin/clsnmp to /usr/sbin/clsnmpe

    Start daemon: snmpd

    • Check the linked binary again

    > ls -la /usr/sbin/snmpd /usr/sbin/clsnmpe

    lrwxrwxrwx 1 root system 18 Nov 09 14:49 /usr/sbin/snmpd -> /usr/sbin/snmpdv3e

    lrwxrwxrwx 1 root system 18 Nov 09 14:49 /usr/sbin/clsnmp -> /usr/sbin/clsnmpe

    • On the solarwinds console, retry "list resources" on the node to confirm all is ok again

     

    Sources

    - IBM Doco

    -- IBM SNMPv3 Documentation [http://www-01.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.networkcomm/snmpv3_intro.htm]

    -- Creating users in SNMPv3 in AIX [http://www-01.ibm.com/support/knowledgecenter/api/content/ssw_aix_71/com.ibm.aix.networkcomm/HT_commadmn_create_snmpv3_user.htm#create_snmpv3_user]

    -- AIX 7.1 snmpdv3.conf file setup [http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.files/snmpdv3.conf.htm?lang=en]

    -- IBM SNMP key generation [http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/com.ibm.aix.commadmn/doc/commadmndita/snmpv3_genkeys.htm]

    -- Using the clsnmp command [http://www-01.ibm.com/support/knowledgecenter/ssw_aix_53/com.ibm.aix.cmds/doc/aixcmds1/clsnmp.htm?cp=ssw_aix_53%2F1-2-0-2-122&lang=en]

    --- Useful to test an snmp configuration as this can interrogate your snmp server to confirm its working ok.

    - Solar Winds Doco

    -- Solarwinds SNMPV3 Implementation Guide [http://www.solarwinds.com/support/Orion/docs/Implementing_SNMPv3r1.pdf]

    -- Managing SNMP credentials guidelines [http://www.solarwinds.com/netperfmon/solarwinds/wwhelp/wwhimpl/js/html/wwhelp.htm]

    - Solar Winds Forums

    -- Example SNMPv3 Configuration on AIX from Solarwinds forum [https://thwack.solarwinds.com/message/249258]

    -- Help with SNMPv3.1 on AIX [http://thwack.solarwinds.com/thread/42695]

    -- Using SNMPv3.1 on AIX [http://thwack.solarwinds.com/thread/36507]

    -- Configuring 3rd party Net-SNMP [http://thwack.solarwinds.com/thread/19323]

    -- AIX Specific mods [http://thwack.solarwinds.com/community/application-and-server_tht/server-and-application-monitor/content?filterID=content~category[application-monitor-templates]&filterID=content~objecttype~objecttype[document]&query=unix]

    - Make AIX a Solarwinds client on snmpv3 [http://odme.blogspot.com.au/2012/09/make-aix-solarwinds-client-on-snmpv3.html]

    - Configuring AIX snmpd for MIB subagent access [http://odme.blogspot.com.au/2012/09/snmpdv3-wont-talk-to-mib-subagents.html]

    -- The aixmbid, snmpmibd and similar subsystems depend on snmp to talk to each other! Here's the settings they use to do so.

    -- Very detailed setup of the AIX snmpd for MIB subagent config including extra mib defaultviews [http://forums.cacti.net/viewtopic.php?t=19040]

    - Nagios setup for snmp [http://nagios.frank4dd.com/howto/aix-snmp-setup.htm]

    -- Examples on using snmpwalk

    -- Example snmpdv3.conf for Nagios with highlights for nagios specifics

    - [http://lparbox.com/how-to/powerha-cluster/21]

    - [http://www.mcpressonline.com/system-administration/techtip-monitoring-aix-with-snmp.html]

    - Using snmpwalk [http://www.net-snmp.org/tutorial/tutorial-5/commands/snmpv3.html]