1 Reply Latest reply on Jun 15, 2018 12:06 PM by bkyle

    Problem with Command issuing on Cisco ASA since update to 3.11.3

    apo14

      Hey there,

       

      since the last update im having several problems, one particular with command issuing and echo timeout i guess.

       

      While issuing the following commands in a job

       

      sh version

      failover exec mate sh vers

      show inventory

      failover exec mate sh inventory

      show int | grep error

      sh access-list outside-access-out | i line 1 extended

      sh access-list outside_access_out | i line 1 extended

      sh failover

       

       

      2 of about 150 devices are aborting due to command error and just showing a single line of the first command in the file:

       

      asa-emft-oph# sh version

       

      the same when issuing just "sh run"

       

       

      Here the debug output for those 2 devices:

      Device1:

      <W-10:37:50>ssh -2 -l nocadmin XXXXXXXXX

      <R-10:37:50>ssh -2 -l nocadmin XXXXXXXXXXX

      <W-10:37:50>[13]

      <R-10:37:50>[13][10]

      <R-10:37:50>nocadmin@XXXXXXX's password:

      <W-10:37:50>XXXXXXXXXXXXXXXXX[13]

      <R-10:37:50>[13][10]

      <R-10:37:51>Type help or '?' for a list of available commands.[13][10][13]asa-emft-oph#

      <W-10:37:51>[13]

      <W-10:37:51>[13]

      <R-10:37:51>[13][10][13]asa-emft-oph#

      <W-10:37:51>[13]

      <R-10:37:51>[13][10][13]asa-emft-oph#

      <W-10:37:51>sh version

      <R-10:37:51>[13][10][13]asa-emft-oph#

      <R-10:37:51>sh version

      <W-10:37:51>[13]

      <W-10:37:52>failover exec mate sh vers

      <R-10:37:52>[13][10][13][10]Cisco Adaptive Security Appliance Software Version 9.1(7)23 [13][10]Device Manager Version 7.8(1)[13][10][13][10]Compiled on Thu 01-Feb-18 23:08 by builders[13][10]System image file is "disk0:/asa917-23-k8.bin"[13][10]Config file at boot was "disk0:/asa-emft-oph.cfg"[13][10][13][10]asa-emft-oph up 125 days 4 hours[13][10][13][10]Hardware:   ASA5520, 2048 MB RAM, CPU Pentium 4 Celeron 2000 MHz,[13][10]Internal ATA Compact Flash, 256MB[13][10]Slot 1: ATA Compact Flash, 512MB[13][10]BIOS Flash M50FW016 @ 0xfff00000, 2048KB[13][10][13][10]Encryption hardware device : Cisco ASA-55xx on-board accelerator (revision 0x0)[13][10]                             Boot microcode        : CN1000-MC-BOOT-2.00 [13][10]                             SSL/IKE microcode     : CNlite-MC-SSLm-PLUS-2.08[13][10]                             IPSec microcode       : CNlite-MC-IPSECm-MAIN-2.09[13][10]                             Number of accelerators: 1[13][10][13][10] 0: Ext: GigabitEthernet0/0  : address is f866.f2c4.a4cc, irq 9[13][10] 1: Ext: GigabitEthernet0/1  : address is f866.f2c4.a4cd, irq 9[13][10] 2: Ext: GigabitEthernet0/2  : address is f866.f2c4.a4ce, irq 9[13][10] 3: Ext: GigabitEthernet0/3  : address is f866.f2c4.a4cf, irq 9[13][10]<--- More --->

      <R-10:37:52>[13]              [13] 4: Ext: Management0/0       : address is f866.f2c4.a4cb, irq 11[13][10] 5: Int: Internal-Data0/0    : address is 0000.0001.0002, irq 11[13][10] 6: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 5[13][10][13][10]Licensed features for this platform:[13][10]Maximum Physical Interfaces       : Unlimited      perpetual[13][10]Maximum VLANs                     : 150            perpetual[13][10]Inside Hosts                      : Unlimited      perpetual[13][10]Failover                          : Active/Active  perpetual[13][10]Encryption-DES                    : Enabled        perpetual[13][10]Encryption-3DES-AES               : Enabled        perpetual[13][10]Security Contexts                 : 2              perpetual[13][10]GTP/GPRS                          : Disabled       perpetual[13][10]AnyConnect Premium Peers          : 2              perpetual[13][10]AnyConnect Essentials             : Disabled       perpetual[13][10]Other VPN Peers                   : 750            perpetual[13][10]Total VPN Peers                   : 750            perpetual[13][10]Shared License                    : Disabled       perpetual[13][10]AnyConnect for Mobile             : Disabled       perpetual[13][10]AnyConnect for Cisco VPN Phone    : Disabled       perpetual[13][10]Advanced Endpoint Assessment      : Disabled       perpetual[13][10]UC Phone Proxy Sessions           : 2              perpetual[13][10]Total UC Proxy Sessions           : 2              perpetual[13][10]Botnet Traffic Filter             : Disabled       perpetual[13][10]<--- More --->[13]              [13]Intercompany Media Engine         : Disabled       perpetual[13][10]Cluster                           : Disabled       perpetual[13][10][13][10]This platform has an ASA 5520 VPN Plus license.[13][10][13][10]Serial Number: JMX1447L037[13][10]Running Permanent Activation Key: 0xd827ec71 0x88d7e59b 0x9c212148 0x9d108494 0x8f201181 [13][10]

      <R-10:37:53>Configuration register is 0x1[13][10]Configuration last modified by admin at 13:33:40.297 CEDT Wed Jun 13 2018[13][10][13]asa-emft-oph# mate sh vers

       

       

      ================================================================================

      WFDRetVal=0. Waiting for: "failoverexecmateshvers"

      WFDBuffer="ciscoadaptivesecurityappliancesoftwareversion9.1(7)23devicemanagerversion7.8(1)compiledonthu01-feb-1823:08bybuilderssystemimagefileis"disk0:/asa917-23-k8.bin"configfileatbootwas"disk0:/asa-emft-oph.cfg"asa-emft-ophup125days4hourshardware:asa5520,2048mbram,cpupentium4celeron2000mhz,internalatacompactflash,256mbslot1:atacompactflash,512mbbiosflashm50fw016@0xfff00000,2048kbencryptionhardwaredevice:ciscoasa-55xxon-boardaccelerator(revision0x0)bootmicrocode:cn1000-mc-boot-2.00ssl/ikemicrocode:cnlite-mc-sslm-plus-2.08ipsecmicrocode:cnlite-mc-ipsecm-main-2.09numberofaccelerators:10:ext:gigabitethernet0/0:addressisf866.f2c4.a4cc,irq91:ext:gigabitethernet0/1:addressisf866.f2c4.a4cd,irq92:ext:gigabitethernet0/2:addressisf866.f2c4.a4ce,irq93:ext:gigabitethernet0/3:addressisf866.f2c4.a4cf,irq9<---more--->4:ext:management0/0:addressisf866.f2c4.a4cb,irq115:int:internal-data0/0:addressis0000.0001.0002,irq116:int:internal-control0/0:addressis0000.0001.0001,irq5licensedfeaturesforthisplatform:maximumphysicalinterfaces:unlimitedperpetualmaximumvlans:150perpetualinsidehosts:unlimitedperpetualfailover:active/activeperpetualencryption-des:enabledperpetualencryption-3des-aes:enabledperpetualsecuritycontexts:2perpetualgtp/gprs:disabledperpetualanyconnectpremiumpeers:2perpetualanyconnectessentials:disabledperpetualothervpnpeers:750perpetualtotalvpnpeers:750perpetualsharedlicense:disabledperpetualanyconnectformobile:disabledperpetualanyconnectforciscovpnphone:disabledperpetualadvancedendpointassessment:disabledperpetualucphoneproxysessions:2perpetualtotalucproxysessions:2perpetualbotnettrafficfilter:disabledperpetual<---more--->intercompanymediaengine:disabledperpetualcluster:disabledperpetualthisplatformhasanasa5520vpnpluslicense.serialnumber:jmx1447l037runningpermanentactivationkey:0xd827ec710x88d7e59b0x9c2121480x9d1084940x8f201181configurationregisteris0x1configurationlastmodifiedbyadminat13:33:40.297cedtwedjun132018asa-emft-oph#mateshvers"

      ================================================================================

      <W-10:38:23>[13]

      <R-10:38:23>[13][10]               ^[13][10]ERROR: % Invalid input detected at '^' marker.[13][10][13]asa-emft-oph#

      <W-10:38:23>[13]

      <R-10:38:23>[13][10][13]asa-emft-oph#

      <W-10:38:23>disable

      <R-10:38:24>disable

      <W-10:38:24>[13]

      <R-10:38:24>[13][10][13]asa-emft-oph>

      <W-10:38:24>[13]

      <R-10:38:24>[13][10][13]asa-emft-oph>

      <W-10:38:24>exit[13]

      <D 10:38:24>

      <SCRIPT VALUES>

      <HOSTNAME="asa-emft-oph">

      <PROMPT VTY="asa-emft-oph>">

      <PROMPT ENABLE="asa-emft-oph#">

      <PROMPT CONFIG="asa-emft-oph(">

       

       

      Device2:

      <W-10:37:50>ssh -2 -l nocadmin XXXXXXXX

      <R-10:37:50>ssh -2 -l nocadmin XXXXXXXXXX

      <W-10:37:50>[13]

      <R-10:37:50>[13][10]

      <R-10:37:50>Unauthorized access prohibited[13][10]nocadmin@192.44.23.130's password:

      <W-10:37:50>XXXXXXXXXXXXXX[13]

      <R-10:37:50>[13][10]

      <R-10:37:51>User nocadmin logged in to asa-iml[13][10]Logins over the last 127 days: 521.  Last login: 10:13:13 CEDT Jun 14 2018 from 153.96.2.16[13][10]Failed logins since the last login: 0.  [13][10]Type help or '?' for a list of available commands.[13][10][13]asa-iml#

      <W-10:37:51>[13]

      <W-10:37:51>[13]

      <R-10:37:51>[13][10][13]asa-iml#

      <W-10:37:51>[13]

      <R-10:37:51>[13][10][13]asa-iml#

      <W-10:37:51>sh version

      <R-10:37:51>[13][10][13]asa-iml#

      <R-10:37:51>sh version

      <W-10:37:51>[13]

      <W-10:37:51>failover exec mate sh vers

      <R-10:37:51>[13][10][13][10]Cisco Adaptive Security Appliance Software Version 9.8(2)20 [13][10]Firepower Extensible Operating System Version 2.2(2.63)[13][10]Device Manager Version 7.8(1)[13][10][13][10]Compiled on Fri 02-Feb-18 06:18 PST by builders[13][10]System image file is "disk0:/asa982-20-smp-k8.bin"[13][10]Config file at boot was "disk0:/asa-iml.cfg"[13][10][13][10]asa-iml up 126 days 19 hours[13][10]failover cluster up 302 days 16 hours[13][10][13][10]Hardware:   ASA5545, 12288 MB RAM, CPU Lynnfield 2660 MHz, 1 CPU (8 cores)[13][10]            ASA: 6466 MB RAM, 1 CPU (1 core)[13][10]Internal ATA Compact Flash, 8192MB[13][10]BIOS Flash MX25L6445E @ 0xffbb0000, 8192KB[13][10][13][10]Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)[13][10]                             Boot microcode        : CNPx-MC-BOOT-2.00[13][10]                             SSL/IKE microcode     : CNPx-MC-SSL-SB-PLUS-0005[13][10]                             IPSec microcode       : CNPx-MC-IPSEC-MAIN-0026[13][10]                             Number of accelerators: 1[13][10]Baseboard Management Controller (revision 0x1) Firmware Version: 2.4[13][10][13][10][13][10]<--- More --->

      <R-10:37:51>[13]              [13] 0: Int: Internal-Data0/0    : address is 0027.e3e4.0220, irq 11[13][10] 1: Ext: GigabitEthernet0/0  : address is 0027.e3e4.0225, irq 5[13][10] 2: Ext: GigabitEthernet0/1  : address is 0027.e3e4.0221, irq 5[13][10] 3: Ext: GigabitEthernet0/2  : address is 0027.e3e4.0226, irq 10[13][10] 4: Ext: GigabitEthernet0/3  : address is 0027.e3e4.0222, irq 10[13][10] 5: Ext: GigabitEthernet0/4  : address is 0027.e3e4.0227, irq 5[13][10] 6: Ext: GigabitEthernet0/5  : address is 0027.e3e4.0223, irq 5[13][10] 7: Ext: GigabitEthernet0/6  : address is 0027.e3e4.0228, irq 10[13][10] 8: Ext: GigabitEthernet0/7  : address is 0027.e3e4.0224, irq 10[13][10] 9: Int: Internal-Data0/1    : address is 0000.0001.0002, irq 0[13][10]10: Int: Internal-Control0/0 : address is 0000.0001.0001, irq 0[13][10]11: Int: Internal-Data0/2    : address is 0000.0001.0003, irq 0[13][10]12: Ext: Management0/0       : address is 0027.e3e4.0220, irq 0[13][10]13: Int: Internal-Data0/3    : address is 0000.0100.0001, irq 0[13][10][13][10]Licensed features for this platform:[13][10]Maximum Physical Interfaces       : Unlimited      perpetual[13][10]Maximum VLANs                     : 300            perpetual[13][10]Inside Hosts                      : Unlimited      perpetual[13][10]Failover                          : Active/Active  perpetual[13][10]Encryption-DES                    : Enabled        perpetual[13][10]Encryption-3DES-AES               : Enabled        perpetual[13][10]Security Contexts                 : 2              perpetual[13][10]Carrier                           : Disabled       perpetual[13][10]<--- More --->[13]              [13]AnyConnect Premium Peers          : 2500           perpetual[13][10]AnyConnect Essentials             : Disabled       perpetual[13][10]Other VPN Peers                   : 2500           perpetual[13][10]Total VPN Peers                   : 2500           perpetual[13][10]AnyConnect for Mobile             : Enabled        perpetual[13][10]AnyConnect for Cisco VPN Phone    : Enabled        perpetual[13][10]Advanced Endpoint Assessment      : Enabled        perpetual[13][10]Shared License                    : Disabled       perpetual[13][10]Total TLS Proxy Sessions          : 2              perpetual[13][10]Botnet Traffic Filter             : Disabled       perpetual[13][10]IPS Module                        : Disabled       perpetual[13][10]Cluster                           : Enabled        perpetual[13][10]Cluster Members                   : 2              perpetual[13][10][13][10]This platform has an ASA5545 VPN Premium license.[13][10][13][10][13][10]Failover cluster licensed features for this platform:[13][10]Maximum Physical Interfaces       : Unlimited      perpetual[13][10]Maximum VLANs                     : 300            perpetual[13][10]Inside Hosts                      : Unlimited      perpetual[13][10]Failover                          : Active/Active  perpetual[13][10]Encryption-DES                    : Enabled        perpetual[13][10]Encryption-3DES-AES               : Enabled        perpetual[13][10]<--- More --->

      <R-10:37:51>[13]              [13]Security Contexts                 : 4              perpetual[13][10]Carrier                           : Disabled       perpetual[13][10]AnyConnect Premium Peers          : 2500           perpetual[13][10]AnyConnect Essentials             : Disabled       perpetual[13][10]Other VPN Peers                   : 2500           perpetual[13][10]Total VPN Peers                   : 2500           perpetual[13][10]AnyConnect for Mobile             : Enabled        perpetual[13][10]AnyConnect for Cisco VPN Phone    : Enabled        perpetual[13][10]Advanced Endpoint Assessment      : Enabled        perpetual[13][10]Shared License                    : Disabled       perpetual[13][10]Total TLS Proxy Sessions          : 4              perpetual[13][10]Botnet Traffic Filter             : Disabled       perpetual[13][10]IPS Module                        : Disabled       perpetual[13][10]Cluster                           : Enabled        perpetual[13][10][13][10]This platform has an ASA5545 VPN Premium license.[13][10][13][10]Serial Number: FCH21147VP5[13][10]Running Permanent Activation Key: 0x752ec867 0x24c86c51 0x11727950 0xda9cf0d8 0xc01cf2bc [13][10]

      <R-10:37:52>Configuration register is 0x1[13][10][13][10]Image type          : Release[13][10]Key version         : A[13][10][13][10]<--- More --->[13]              [13]Configuration last modified by root at 09:56:02.322 CEDT Wed Jun 13 2018[13][10][13]asa-iml# vers

       

       

      ================================================================================

      WFDRetVal=0. Waiting for: "failoverexecmateshvers"

      WFDBuffer="ciscoadaptivesecurityappliancesoftwareversion9.8(2)20firepowerextensibleoperatingsystemversion2.2(2.63)devicemanagerversion7.8(1)compiledonfri02-feb-1806:18pstbybuilderssystemimagefileis"disk0:/asa982-20-smp-k8.bin"configfileatbootwas"disk0:/asa-iml.cfg"asa-imlup126days19hoursfailoverclusterup302days16hourshardware:asa5545,12288mbram,cpulynnfield2660mhz,1cpu(8cores)asa:6466mbram,1cpu(1core)internalatacompactflash,8192mbbiosflashmx25l6445e@0xffbb0000,8192kbencryptionhardwaredevice:ciscoasacryptoon-boardaccelerator(revision0x1)bootmicrocode:cnpx-mc-boot-2.00ssl/ikemicrocode:cnpx-mc-ssl-sb-plus-0005ipsecmicrocode:cnpx-mc-ipsec-main-0026numberofaccelerators:1baseboardmanagementcontroller(revision0x1)firmwareversion:2.4<---more--->0:int:internal-data0/0:addressis0027.e3e4.0220,irq111:ext:gigabitethernet0/0:addressis0027.e3e4.0225,irq52:ext:gigabitethernet0/1:addressis0027.e3e4.0221,irq53:ext:gigabitethernet0/2:addressis0027.e3e4.0226,irq104:ext:gigabitethernet0/3:addressis0027.e3e4.0222,irq105:ext:gigabitethernet0/4:addressis0027.e3e4.0227,irq56:ext:gigabitethernet0/5:addressis0027.e3e4.0223,irq57:ext:gigabitethernet0/6:addressis0027.e3e4.0228,irq108:ext:gigabitethernet0/7:addressis0027.e3e4.0224,irq109:int:internal-data0/1:addressis0000.0001.0002,irq010:int:internal-control0/0:addressis0000.0001.0001,irq011:int:internal-data0/2:addressis0000.0001.0003,irq012:ext:management0/0:addressis0027.e3e4.0220,irq013:int:internal-data0/3:addressis0000.0100.0001,irq0licensedfeaturesforthisplatform:maximumphysicalinterfaces:unlimitedperpetualmaximumvlans:300perpetualinsidehosts:unlimitedperpetualfailover:active/activeperpetualencryption-des:enabledperpetualencryption-3des-aes:enabledperpetualsecuritycontexts:2perpetualcarrier:disabledperpetual<---more--->anyconnectpremiumpeers:2500perpetualanyconnectessentials:disabledperpetualothervpnpeers:2500perpetualtotalvpnpeers:2500perpetualanyconnectformobile:enabledperpetualanyconnectforciscovpnphone:enabledperpetualadvancedendpointassessment:enabledperpetualsharedlicense:disabledperpetualtotaltlsproxysessions:2perpetualbotnettrafficfilter:disabledperpetualipsmodule:disabledperpetualcluster:enabledperpetualclustermembers:2perpetualthisplatformhasanasa5545vpnpremiumlicense.failoverclusterlicensedfeaturesforthisplatform:maximumphysicalinterfaces:unlimitedperpetualmaximumvlans:300perpetualinsidehosts:unlimitedperpetualfailover:active/activeperpetualencryption-des:enabledperpetualencryption-3des-aes:enabledperpetual<---more--->securitycontexts:4perpetualcarrier:disabledperpetualanyconnectpremiumpeers:2500perpetualanyconnectessentials:disabledperpetualothervpnpeers:2500perpetualtotalvpnpeers:2500perpetualanyconnectformobile:enabledperpetualanyconnectforciscovpnphone:enabledperpetualadvancedendpointassessment:enabledperpetualsharedlicense:disabledperpetualtotaltlsproxysessions:4perpetualbotnettrafficfilter:disabledperpetualipsmodule:disabledperpetualcluster:enabledperpetualthisplatformhasanasa5545vpnpremiumlicense.serialnumber:fch21147vp5runningpermanentactivationkey:0x752ec8670x24c86c510x117279500xda9cf0d80xc01cf2bcconfigurationregisteris0x1imagetype:releasekeyversion:a<---more--->configurationlastmodifiedbyrootat09:56:02.322cedtwedjun132018asa-iml#vers"

      ================================================================================

      <W-10:38:22>[13]

      <R-10:38:22>[13][10]            ^[13][10]ERROR: % Invalid input detected at '^' marker.[13][10][13]asa-iml#

      <W-10:38:22>[13]

      <R-10:38:22>[13][10][13]asa-iml#

      <W-10:38:22>disable

      <R-10:38:22>disable

      <W-10:38:22>[13]

      <R-10:38:22>[13][10][13]asa-iml>

      <W-10:38:22>[13]

      <R-10:38:22>[13][10][13]asa-iml>

      <W-10:38:23>exit[13]

      <D 10:38:23>

      <SCRIPT VALUES>

      <HOSTNAME="asa-iml">

      <PROMPT VTY="asa-iml>">

      <PROMPT ENABLE="asa-iml#">

      <PROMPT CONFIG="asa-iml(">

       

       

       

      The Devices are accessed with a jumphost based on linux which never made any problems before, and does not interfere with alle the other ASAs where its working, so i guess this isnt the problem.

       

      Any suggestions?