10 Replies Latest reply on Jan 9, 2015 4:45 AM by martynthomas

    Backing up a PFSense Firewall over SSH using Generic.Device

    martynthomas

      Hi chaps,

       

      I'm struggling a little to get the Generic.Device with Variations to work correctly when trying to back up my PFSense Firewall.

       

      Here is the contents of my variations file:

       

      DEVICE_PRIVILEGEDPROMPT = "):"

      DEVICE_INVALIDCOMMAND = "% Command not found."

      COMMAND_DISABLEPAGING = ""

      COMMAND_ENABLEPAGING = ""

      COMMAND_RUNNINGCONFIG = "cat /conf/config.xml"

      COMMAND_STARTUPCONFIG = "cat /conf/config.xml"

      COMMAND_DISCONNECT = "exit"

      RESPONSE_STRIP_VT100ESC = "1"

      RESPONSE_STRIP_ANSICHARS = "1"

      RESPONSE_STRIP_NULLS = "1"

       

       

      Sanitised Info Log:

       

      2015-01-06 10:58:54    3-Info    0    CatTools Service    Performing activity - Run Now

      2015-01-06 10:58:54    3-Info    0    CatTools Service    Loading activity: Device.Backup.Running Config - MT PFSense. Schd: 4

      2015-01-06 10:58:54    4-Debug    0    CatTools Service    Marshaller - Running script. Device: Firewall

      2015-01-06 10:58:54    3-Info    1    Firewall    Loading variations for  C:\Program Files (x86)\CatTools3\Variations\C4L_Management_FW.txt

      2015-01-06 10:58:54    3-Info    1    Firewall    Variations function found

      2015-01-06 10:58:54    4-Debug    1    Firewall    SSH Fingerprint: ################################

      2015-01-06 10:58:55    4-Debug    1    Firewall    Connected to 192.168.1.254

      2015-01-06 10:58:55    4-Debug    1    Firewall    Login Generic Device: Firewall

      2015-01-06 10:58:55    4-Debug    1    Firewall    Waiting for command prompt

      2015-01-06 10:58:55    4-Debug    1    Firewall    DeviceHostnameID: [2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1

      2015-01-06 10:58:55    4-Debug    1    Firewall    Login to Firewall was successful

      2015-01-06 10:58:55    4-Debug    1    Firewall    Skipping enter enable mode as we are already in enable mode

      2015-01-06 10:58:55    4-Debug    1    Firewall    Waiting for an echo of cat /conf/config.xml command

      2015-01-06 10:59:25    1-Error    1    Firewall    Did not receive echo of cat /conf/config.xml command

      2015-01-06 10:59:25    4-Debug    1    Firewall    Did not receive echo of cat /conf/config.xml

      2015-01-06 10:59:25    3-Info    1    Firewall    Backup Running Config results: Failed

      2015-01-06 11:00:26    4-Debug    1    Firewall    Disconnecting from Firewall

      2015-01-06 11:00:26    4-Debug    1    Firewall    Disconnected from 192.168.1.254

      2015-01-06 11:00:26    3-Info    0    CatTools Service    Stopping Activity.

      2015-01-06 11:00:26    3-Info    0    CatTools Service    All threads have finished. Now processing results...

      2015-01-06 11:00:26    3-Info    0    CatTools Service    Run Now activity has completed

       

       

      Debug:

       

      I've snipped the output of the config on line 21 to remove anything sensitive but you get the idea.

       

      <NEWSESSION CatTools 3.10.0 06/01/2015 10:58:54>

      <PROTOCOL=SSH2>

      <DEVICE TYPE=Generic.Device>

      <ACTIVITY TYPE=Device.Backup.Running Config>

      <ACTIVITY SCRIPT=C:\Program Files (x86)\CatTools3\Scripts\Client.Device.Backup.Running Config.txt>

      <USERS NAME FOR DEVICE=C4L Management FW>

      <C OK 10:58:55>

      <R-10:58:55>Last login: Tue Jan  6 10:51:12 2015 from 192.168.10.35[13][13][10]Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994[13][10][09]The Regents of the University of California.  All rights reserved.[13][10][13][10][2.1.5-RELEASE][cattools@bgmcfw2.bchosting.co.uk]/home/cattools(1):

      <W-10:58:55>[13]

      <R-10:58:55>[13][13][10][2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1):

      <W-10:58:55>[13]

      <R-10:58:55>[13][13][10][2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1):

      <W-10:58:55>cat /conf/config.xml

      <R-10:58:55>cat /conf/co [08]nfig.xml

       

      ================================================================================

      WFDRetVal=0. Waiting for: "cat /conf/config.xml"

      WFDBuffer="cat /conf/co [08]nfig.xml"

      ================================================================================

      <W-10:59:25>[13]

      <R-10:59:25>[13][13][10]<?xml version="1.0"?>[13][10]<pfsense>[13][10][09]<version>10.1</version>[13][10][09]<lastchange/>[13][10][09]<theme>pfsense_ng</theme>[13][10][09]<sysctl>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Enable mounting the FS read only with more checks.]]></descr>[13][10][09][09][09]<tunable>vfs.forcesync</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Disable the pf ftp proxy handler.]]></descr>[13][10][09][09][09]<tunable>debug.pfftpproxy</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Increase UFS read-ahead speeds to match current state of hard drives and NCQ. More information here: http://ivoras.sharanet.org/blog/tree/2010-11-19.ufs-read-ahead.html]]></descr>[13][10][09][09][09]<tunable>vfs.read_max</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Set the ephemeral port range to be lower.]]></descr>[13][10][09][09][09]<tunable>net.inet.ip.portrange.first</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Drop packets to closed TCP ports without returning a RST]]></descr>[13][10][09][09][09]<tunable>net.inet.tcp.blackhole</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Do not send ICMP port unreachable messages for closed UDP ports]]></descr>[13][10][09][09][09]<tunable>net.inet.udp.blackhole</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Randomize the ID field in IP packets (default is 0: sequential IP IDs)]]></descr>[13][10][09][09][09]<tunable>net.inet.ip.random_id</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Drop SYN-FIN packets (breaks RFC1379, but nobody uses it anyway)]]></descr>[13][10][09][09][09]<tunable>net.inet.tcp.drop_synfin</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Enable sending IPv4 redirects]]></descr>[13][10][09][09][09]<tunable>net.inet.ip.redirect</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Enable sending IPv6 redirects]]></descr>[13][10][09][09][09]<tunable>net.inet6.ip6.redirect</tunable>[13][10][09][09][09]<value>default</value>[13][10][09][09]</item>[13][10][09][09]<item>[13][10][09][09][09]<descr><![CDATA[Enable privacy settings for IPv6 (RFC 4941)]]></descr>[13][10][09][09][09]

      <snip>
      </pfsense>[13][10][2.1.5-release][cattools@firewall.local]/home/cattools(2): "

      ================================================================================

      WFMDRetVal=1 Waiting for: "[2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1>"

      WFMDRetVal=2 Waiting for: "[2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1):"

      WFMDRetVal=3 Waiting for: "[2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1("

      WFMDRetVal=4 Waiting for: "(config)"

      WFMDBuffer="[13][13][10][2.1.5-release][cattools@firewall.local]/home/cattools(2): "

      ================================================================================

      <W-11:00:26>exit[13]

      <D 11:00:26>

      <SCRIPT VALUES>

      <HOSTNAME="[2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1">

      <PROMPT VTY="[2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1>">

      <PROMPT ENABLE="[2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1):">

      <PROMPT CONFIG="[2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1(">

       

       

      Any thoughts how I can clean up the output so the output/config is retrieved correctly?

       

      Thanks,

       

      Martyn

        • Re: Backing up a PFSense Firewall over SSH using Generic.Device
          bkyle

          Marty,

           

          Please post the contents of a Putty log showing a successful device backup.

            • Re: Backing up a PFSense Firewall over SSH using Generic.Device
              martynthomas

              As requested:

               

              =~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2015.01.06 17:20:48 =~=~=~=~=~=~=~=~=~=~=~=

              login as: cattools

              Using keyboard-interactive authentication.

              Password:

              Last login: Tue Jan  6 17:20:04 2015 from 192.168.30.103 Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994

                  The Regents of the University of California.  All rights reserved.

               

              [0;1;33m[ [0;1;37m2.1.5-RELEASE [0;1;33m] [0;1;33m [1m[ [0;1;37mcattools [0;1;31m@ [0;1;37mfirewall.local [0;1;33m] [0;1;32m [m/home/cattools [0;1;33m( [0;1;37m1 [0;1;33m) [0;1;36m [0;1;31m: [0;0;0m cat /conf/co  nfig.xml <?xml version="1.0"?>

              <pfsense>

                  <version>10.1</version>

                  <lastchange/>

                  <theme>pfsense_ng</theme>

                  <sysctl>

                      <item>

                          <descr><![CDATA[Enable mounting the FS read only with more checks.]]></descr>

                          <tunable>vfs.forcesync</tunable>

                          <value>default</value>

                      </item>

                      <item>

                          <descr><![CDATA[Disable the pf ftp proxy handler.]]></descr>

                          <tunable>debug.pfftpproxy</tunable>

                          <value>default</value>

                      </item>

                      <item>

                          <descr><![CDATA[Increase UFS read-ahead speeds to match current state of hard drives and NCQ. More information here: http://ivoras.sharanet.org/blog/tree/2010-11-19.ufs-read-ahead.html]]></descr>

                          <tunable>vfs.read_max</tunable>

                          <value>default</value>

                      </item>

                     

              </pfsense>

              [0;1;33m[ [0;1;37m2.1.5-RELEASE [0;1;33m] [0;1;33m [1m[ [0;1;37mcattools [0;1;31m@ [0;1;37mfirewall.local [0;1;33m] [0;1;32m [m/home/cattools [0;1;33m( [0;1;37m2 [0;1;33m) [0;1;36m [0;1;31m: [0;0;0m exit logout

            • Re: Backing up a PFSense Firewall over SSH using Generic.Device
              Fodome

              Hello Martyn,

               

              The problem I am seeing is that the original prompt is:


              [2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1):


              It then changes to the following after it receives the configuration information:


              [2.1.5-release][cattools@firewall.local]/home/cattools(2):


              As CatTools is looking for "[2.1.5-RELEASE][cattools@firewall.local]/home/cattools(1" to ensure that the command completed and it does not see it, it assumes the command failed at some point.


              In order to fix this, open the Variations for the Device, go to the "Full Prompts" tab and set the Enable Prompt to "firewall.local]/home/cattools".  This way, every time CatTools sees "firewall.local]/home/cattools", it will assume that it is at an Enable prompt.


              Hopefully this helps.


              Sincerely,


              Chris Foley (a.k.a. Fodome)

              • Re: Backing up a PFSense Firewall over SSH using Generic.Device
                Fodome

                Martyn,

                 

                I've just noticed that your output had 2 errors.  The other one is that the Echo of the command does not match the command.

                 

                -Here we see CatTools sending the command:

                <W-10:58:55>cat /conf/config.xml

                 

                -Here we see the target device responding with an Echo of the command:

                <R-10:58:55>cat /conf/co [08]nfig.xml

                 

                The problem is that the echo contains an added backspace character of [08].  As a result, CatTools believes the command has failed.

                 

                The following variables will strip out some control characters such as Nulls, CRs and LFs but not backspaces.

                 

                RESPONSE_STRIP_VT100ESC = "1"

                RESPONSE_STRIP_ANSICHARS = "1"

                RESPONSE_STRIP_NULLS = "1"

                 

                Having all that said, the only way to fix this would be to configure the device to not add a backspace character in the echo of the command.

                 

                Sincerely,

                Fodome

                  • Re: Backing up a PFSense Firewall over SSH using Generic.Device
                    martynthomas

                    Hi Chris,

                     

                    Thanks for responding again.

                     

                    Any idea how I might do that? As you probably know, PFSense is based on FreeBSD.

                     

                    Thanks,

                     

                    Martyn

                      • Re: Backing up a PFSense Firewall over SSH using Generic.Device
                        martynthomas

                        For anyone else with this issue, i've managed to work around the issue by changing from the default shell to /bin/sh and then executing the command to dump the config.


                        Final variation file config for anyone else that needs it:

                         

                        DEVICE_PRIVILEGEDPROMPT = "):"

                        DEVICE_INVALIDCOMMAND = "% Command not found."

                        COMMAND_ENTERENABLEMODE = "/bin/sh"

                        COMMAND_EXITENABLEMODE = ""

                        COMMAND_DISABLEPAGING = ""

                        COMMAND_ENABLEPAGING = ""

                        COMMAND_RUNNINGCONFIG = "cat /conf/config.xml"

                        COMMAND_STARTUPCONFIG = "cat /conf/config.xml"

                        COMMAND_DISCONNECT = "exit"

                        RESPONSE_STRIP_VT100ESC = "1"

                        RESPONSE_STRIP_ANSICHARS = "1"

                        RESPONSE_STRIP_NULLS = "1"

                        FULL_VTYPROMPT = "/home/cattools"

                        FULL_ENABLEPROMPT = "$"

                         

                        It's also worth creating a dedicated login for CatTools to avoid having to exit from the PFSense menu system that is presented when logging in with 'admin'.

                         

                        Cheers,

                         

                        Martyn

                        1 of 1 people found this helpful