31 Replies Latest reply on Sep 11, 2012 9:42 PM by Steve Welsh

    Device.Backup.Running Config not comparing files

    jfitzell

      I have been running CatTools 3.4 Free on an XP machine for some time and found it brilliant.

      I've recently replaced this machine with a Win 7 x64 machine and installed CatTools 3.6 Free.  Everything is working correctly except for the compare running configs which tells me that there is no change even when there is.  This is occurring for several Cisco and Juniper models and they worked fine under 3.4 (which unfortunately I don't have anymore).

      I dont' know if it's a cause or a result but the Dated Configs folder remains empty and does not have any history of the changed configs either.  The version of the running config in the Config directory is always correct (ie it updates correctly when the activity runs) it's just the comparison and Dated Config that fails (or never occurs).

      Please help, this is exactly what I used CatTools for.  I've tried a fresh install and created new activities to no avail.

      Thanks,
      J

        • Re: Device.Backup.Running Config not comparing files
          jfitzell

          I seem to have resolved my problem, although I can't quite explain why.

          I'd already tried reinstalling, starting a new database etc, however I hadn't tried manually configuring a new device (I'd been importing from an exported device list).  Having setup a new device I successfully compared it to it's old config.  I then ran my other activity (unchanged) and now it's successfully comparing configs too.

          Weird, but it works and I'm happy.

          • Re: Device.Backup.Running Config not comparing files
            jfitzell

            Stranger and stranger.  After deleting the new device the original activity went back to failing to compare configs.  I guess there's something wrong with the imported devices that is somehow resolved when there's a manually created one.

            For now I guess I'll just have to re-create all my devices instead of importing them.

              • Re: Device.Backup.Running Config not comparing files
                Steve Welsh

                Hi jfitzell,

                Was the failing to compare configs an intermittent issue or was it something which was occurring with every backup activity run?

                Kind regards,

                Steve

                 

                  • Re: Device.Backup.Running Config not comparing files
                    jfitzell

                    Hi Steve,

                    It was occurring with every run of the backup activity.  Creating a new device manually resolved the issue, not just for the newly created device but also for the "original device" that still existed (ie two separate names for the same physical network object).  My memory is it even resolved the issue for unrelated devices that were of the same device type (ie all Cisco ASA's for example) but unfortunately I've now set them all up from scratch and can't confirm.

                    It should be easy to replicate:

                    • Fresh install of CatTools 3.6 (trial in my case, planning to revert to free)
                    • Import backup (or device list) of existing config (in my case, from CatTools 3.4 free)
                    • Devices appear correct, activities appear correct... but that specific activity does not compare backups.

                    Cheers,
                    James

                      • Re: Device.Backup.Running Config not comparing files
                        Steve Welsh

                        Hi James,

                        Thanks for the extra info regarding this issue.  It does sound very strange that adding a new device manually should fix the problem for all the other devices using the same device type.

                        I'm glad you got it working... although adding all devices again manually must have been a pain!

                        I don't suppose you could send me a copy of your device and activity import files, so I can try replicate the problem in our lab?  Let me know and I'll send you an off-line message (as I guess you wouldn't want your device details on a public forum!)

                        Regards,

                        Steve

                          • Re: Device.Backup.Running Config not comparing files

                            I am having a similar issue.  My version (3.6.0) is running on Windows 2008 R2.   This is a fresh install.  When I manually run the device.backup.running conifig activity, the script correctly processes any change - email is sent, and a Dated Configs entry is written.  However, when I let the timer run the job and there is a change to be processed, InfoLog reports No Change and overwrites the current config (e.g. Config.Current.Running.device.txt) and nothing is written to Dated Configs.

                              • Re: Device.Backup.Running Config not comparing files
                                jfitzell

                                Hi craigcraig,

                                Unfortately I am still having the problem too, I've just given up on scheduler and need to run it manually every time I want a comparison of my configs (which I agree defeats the purpose, but I haven't had time to investigate further).

                                Spartacus, the problem seems intermittent but is broken more often than not (like I get an email only once every month or so, despite changes in between).  Unfortunately I'm not able to send you a copy of my device files or my activity, however they are standard Cisco ASA's connected via SSH running the default backup and compare activity.

                                Cheers,
                                J

                                  • Re: Device.Backup.Running Config not comparing files
                                    Steve Welsh

                                    Hi jfitzell,

                                    Can you confirm which Windows O/S are you running CatTools on please?

                                    Thanks,

                                    Steve.

                                      • Re: Device.Backup.Running Config not comparing files
                                        jfitzell

                                        Hi Spartacus,

                                        Windows 7 x64 (Enterprise).

                                        Cheers,
                                        J

                                          • Re: Device.Backup.Running Config not comparing files

                                            What I've done is create an activity so that it creates a new file every time.  Under the options tab for the activity, I modified the "current config file" value to:  

                                            E:\Program Files (x86)\CatTools3\Configs\DailyBackup\Config.Current.Running.%BaseFile%.%DateISO%-%TimeHHMM%.txt

                                             

                                            I have it run once per day via the timer (set to run at 4am).  That seems to work - every day it runs; I get a report stating all the configs are new.  So the timer works, and the comparison engine is being called.  Or so it would seem.

                                            To keep that directory from growing too large, I could write a vb script that cleans up based on file date.  It would be a decent solution for now, although the dated configs would be the holy grail...  :-)

                                              • Re: Device.Backup.Running Config not comparing files
                                                jfitzell

                                                Hi Craig,

                                                Sorry for the delayed response.  I've tested your approach, and I guess it does satisfactorily provide a backup but it defeats the main purpose of using CatTools for me.  I use CatTools to get notified when a configuration has been edited, and when the running-config doesn't match the statup-config.  Both of these require the comparison engine which is where the problem seems to lie.

                                                Unfortunately, after much researching the problem, I've simply stopped using CatTools because I was unable to resolve the issue.

                                                Cheers,
                                                J

                              • Re: Device.Backup.Running Config not comparing files
                                bravoromeocharlie

                                I'm experiencing problems similar, if not identical, to yours. We are using CatTools 3.6.0 (running on Windows Server 2008R2 Enterprise) to back up a number of F5 load balancers. From time to time CatTools will both fail move the old config to the "Dated Configs" directory and fail to indicate there was a change in the activity report e-mail.

                                Investigating this problem I've found that CatTools is able to grab the changed config and store it in "Current Configs" directory, but (for reasons as yet unknown) the comparison (diff) process is failing.

                                  • Re: Device.Backup.Running Config not comparing files
                                    Steve Welsh

                                    bravoromeocharlie and other thread followers:

                                    I worked with a customer recently with regard to this issue but when I went through with him, the diffs appeared to be working perfectly (which is never very helpful when trying to find the cause of an issue).

                                    Customer said: 
                                    Please close this ticket. This issue was resolved by what looks to be some combination of recreating the tasks, enable/disable tasks, and other diagnostic work that was done.

                                    Unfortunately (despite follow-up requests) the customer did not elaborate any further with any specifics in regards to the 'diagnostics work' so we are still not sure as to what ultimately fixed the problem.

                                    However, if you are experiencing the same issue, please raise a support ticket.  Support can then look at setting up a Goto session with development, so we can go through the same steps on your system to see if this resolves it.

                                    bravoromeocharlie - I am especially interested in your case as you say "From time to time CatTools will both fail move the old config to the "Dated Configs" directory..."  When this occurs does CatTools report the config change correctly though for this device? 

                                    Regards,

                                    Steve

                                      • Re: Device.Backup.Running Config not comparing files
                                        bravoromeocharlie

                                        Steve - Thank you for the reply.

                                        At this time we are using the free version of CatTools and don't have a support contract, but I am looking into the option of purchasing the software and support.

                                        In my case CatTools has NOT been reporting the change. I'm not sure how many changes we're missing, I know that in June there were at least two instances where I made a change, which showed up in the "Current Config File" the next day, but no changes were reported in the nightly backup report e-mail, no diff files were generated, and no dated config file was created.

                                          • Re: Device.Backup.Running Config not comparing files
                                            Steve Welsh

                                            Whenever the backup activity (Device.Backup.Running Config) runs, it always updates the 'Current Config File' regardless of whether changes have been detected or not.

                                            Previously you mentioned "From time to time CatTools will both fail move the old config to the "Dated Configs" directory..."  but I'm presuming now you are meaning that this occurs at the same time when CatTools doesn't notify you of the changes too (i.e. short, as it hasn't been able to detect the change, it therefore doesn't date the current config and move into the dated configs folder).

                                            Steve

                                              • Re: Device.Backup.Running Config not comparing files
                                                bravoromeocharlie

                                                Whenever the backup activity (Device.Backup.Running Config) runs, it always updates the 'Current Config File' regardless of whether changes have been detected or not.

                                                Right. I mentioned the 'Curent Config File' because I wanted to make it clear that CatTools wasn't having any problems retrieving the modified config from our network devices.

                                                Previously you mentioned "From time to time CatTools will both fail move the old config to the "Dated Configs" directory..."  but I'm presuming now you are meaning that this occurs at the same time when CatTools doesn't notify you of the changes too (i.e. short, as it hasn't been able to detect the change, it therefore doesn't date the current config and move into the dated configs folder).

                                                Exactly! It appears that the diff process is breaking such that CatTools is unaware there has been a change and is failing to take the expected actions (note that the config has changed in the backup report e-mail, date the current config and copy it to the dated configs folder, etc.).

                                        • Re: Device.Backup.Running Config not comparing files
                                          martens72

                                          I have pretty much the same issue using the Free Version 3.6.0

                                          I am running on 2008R2 and only backing up a handful of devices. We are looking at expanding that to include all our remote facilities; hoewever I need to see a resolution to this issue.

                                          I make changes regurlarly lately to our infrastructure, and none of the changes are detected by Cattools, hence no archived configs for reference.

                                          If there is any information that can be supplied, I would be happy to do so. Unfortunately, I did not run a previous version of Cattools to compare against.

                                            • Re: Device.Backup.Running Config not comparing files
                                              Steve Welsh

                                              There seems to be many different causes and solutions to this problem (Firewall blocking, Anti-Virus, etc.)

                                               

                                              One more you may like to investigate is the security permissions for the diff.exe in the CatTools install folder.

                                              I did some testing today on 2008 R2 changing the security settings for the diff.exe and was able to replicate this issue in that CatTools simply return 'no change' although there have been changes.

                                              You may like to try explicitly allowing full permissions on the diff.exe for the user account CatTools is running under.

                                              Please post back your results.

                                               

                                              Regards,

                                               

                                              Steve

                                                • Re: Device.Backup.Running Config not comparing files
                                                  martens72

                                                  Excellent!

                                                  I went into compatibility settings on the diff.exe application and set it to run as administrator for all users. The initial test detectected changes on all devices, which is something it has not done in the past. We have a set of ASA's that replicate config for failover, and generally Cattools will detect change on one, but not the other.

                                                  I will run another test later, and update the case.

                                                  Thanks again,

                                                  Mike Martens

                                                  • Re: Device.Backup.Running Config not comparing files
                                                    martens72

                                                    Well...it looks like Cattools is still hit or miss. I made some changes on our infrastructure on 3 devices. Cattools logged in, grabbed the configs, and only emailed changes on the secondary ASA. It did not pick up the changes on our switch, or the changes on the primary ASA.

                                                    I will try the looking into the DEP recommendation, but it the diff seems to work on at least one device, so diff.exe should have the appropriate permissions on the system.

                                                      • Re: Device.Backup.Running Config not comparing files
                                                        jfitzell

                                                        Hi martens,

                                                        After working with Steve I discovered that, in my case at least, the Symantec firewall was causing me issues with the diff process running, BUT it was still working some of the time and for some of the devices.  When Symantec was causing problems I encountered exactly the scenario you describe so I can confirm that just because it works some of the time or for some of the devices does not mean that the permissions are correct.

                                                        For me, after messing with security settings in Symantec firewall I got everything working (basically I allowed all traffic to/from the cattools and diff executables).  It makes no sense that this is required since the downloads are occurring and it's the local diff that is failing but this is what resolved it for me.  I've since upgraded my version of Symantec to 12.1 simultaneously removing the Symantec firewall, falling back on the built in Windows 7 firewall, and CatTools continues to work (I didn't need to make the changes in the Windows firewall, in my situation it was only the 3rd party Symantec firewall causing problems).

                                                        Cheers,
                                                        J

                                                          • Re: Device.Backup.Running Config not comparing files
                                                            martens72

                                                            I'll have to see if I can exclude Cattools from the Forefront client security we are using, otherwise all the ports are opened on the Firewall.  Since the 3rd of October, Cattools has been catching the changes.

                                                            The main issue I have is that I get reports indicating changes to only one of two replicated devices. I like Cattools for what it does, and it's great for the ASA, which doesn't have archive feature, so I really want this to work, but we also installed a Linux box recently, and I may start running side by side with RANCID, or other app.

                                                    • Re: Device.Backup.Running Config not comparing files
                                                      Steve Welsh

                                                      This issue was resolved in CatTools v3.7.0.  It related specifically to Windows 7 and Windows Server 2008 R2 file management changes within these O/S.

                                                       

                                                      Prior to v3.7.0 CatTools would re-use the same temporary file for diff reporting, which for Windows 7 and Windows Server 2008 R2 would result in mis-reporting changes due to the file not always being available for update.  However for v3.7.0 unique temporary files are now generated per device which resolves this issue.

                                                       

                                                      Steve