28 Replies Latest reply on Feb 18, 2009 8:34 PM by davidmaltby

    NetFlow  v3.1 Upgrade

    usmc_ed

      Like many NetFlow (NF) users, I have been dreaming  of a day when NF functions properly and efficiently. Seems 3.1 may have solved this, at least so far. I am hesitant in making that statement however, from past experience.

      I have upgraded to v3.1 and noticed modest performance improvements. I can't seem to determine if the upgrade is complete or not. When you install the upgrade, you are informed that due to the vast amounts of changes to the DB, that the upgrade will continue to run in the background. This progress is tracked by the NPM event logs. Applied upgrade yesterday and last reported completion percentage was 36%.

      Has anyone received a 100% complete log or know how to check to see if the upgrade is complete?

        • Re: NetFlow  v3.1 Upgrade
          sotherls

          Yes. Shortly after I did the upgrade I got a completion message however our NetFlow usage is low and I only have one source currently per network.

            • Re: NetFlow  v3.1 Upgrade
              usmc_ed

              I see...

              Entering the 23rd hour… Yes!  23rd hour… and the upgrade continues.

               

              I started to see process event logs again a few hours ago and now we are at 61% complete. I was to the point where I wanted to blow away all NF tables and start from scratch. Luckily SW tech advised to stay with it. Whatever is going on, is working. My DB size has gone from 123GB down to 33GB.

               

            • Re: NetFlow  v3.1 Upgrade
              SamuelB


              Like many NetFlow (NF) users, I have been dreaming  of a day when NF functions properly and efficiently. Seems 3.1 may have solved this, at least so far. I am hesitant in making that statement however, from past experience.

               



              I would just like to add on to this comment and say that I upgraded yesterday and the speed improvements have been HUGE! The nightly maintenance only made things even faster! I appreciate all of the work that went into making NTA faster!

              • Re: NetFlow  v3.1 Upgrade
                PeteF

                Additional:

                I'm seeing my SolarWindsOrion_FG4 file balloon to seemlingly ridiculous sizes compared with the original database I'm upgrading from.

                eg. under NetFlow v3.0 regime, my current Solarwinds Orion Database is around 23GB.

                At the time of writing this post, l'm running NetFlow v3.1 setup, the install is still happening and I'm seeing 4 additional DB's like everyone else, but FG4 is currently at 52GB (and climbing) and the upgrade is still happening!

                Seeing I've only got 100GB free on the drive in total, how large can I expect these DB's to get?

                (Did anyone else see this happen?)

                - Pete

                  • Re: NetFlow  v3.1 Upgrade
                    davidmaltby

                    Filegroup FG4 contains only uncompressed data from new flows that you've received since your upgrade.  In NTA 3.1, the service is now able to capture a lot more flows then it did in 3.0.  Our tests have shown in that in some high stress scenarios, that we were dropping as much as 40% of the flows coming in before they were able to be stored in the database.

                    What is your setting on how much uncompressed data to store?  If you adjust that value down, you'll end up reducing your FG4 size when the next nightly maintance comes around and shrinks your database.

                    Thanks,

                    David

                      • Re: NetFlow  v3.1 Upgrade
                        PeteF

                        The FG4 file finished at 62GB, I understand that data was compressed I just wasn't expecting to need more storage to perform the upgrade but it makes sense after the fact.

                        I've got default values for data retention:

                        Keep uncompressed data for 4 hours 
                        Keep compressed data for 365 days

                        I'm now seeing table migration in the Orion events so things are moving. I'm unable to view NetFlow Traffic Analysis at the moment as I get this error:

                         

                        Orion Website Error

                        An error has occurred with the Orion website.

                        Additional Information

                        System.FormatException: Input string was not in a correct format.
                           at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
                           at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
                           at SolarWinds.Netflow.Reporting.NetflowSettingsDAL.GetSummary2Period()
                           at SolarWinds.Netflow.Reporting.DateTimeBoundaryHelper.RoundAccordingToCollapseInterval(DateTime& dateTime, Boolean roundGreater)
                           at SolarWinds.Netflow.Reporting.DateTimeBoundaryHelper.GetPreviousMinuteBoundary(Int32 index, DateTime oldDate, Boolean roundGreater)
                           at SolarWinds.Netflow.Reporting.TimePeriod.SetRelativePeriod(String PeriodName, String timeField, Int32 timeValue)
                           at SolarWinds.Netflow.Reporting.TimePeriod.ParseName(String PeriodName)
                           at SolarWinds.Netflow.Reporting.TimePeriod..ctor(String PeriodName)
                           at SolarWinds.Netflow.Web.UI.TopXXBase.DataBind()
                           at System.Web.UI.Control.DataBindChildren()
                           at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
                           at System.Web.UI.Control.DataBind()
                           at System.Web.UI.Control.DataBindChildren()
                           at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
                           at System.Web.UI.Control.DataBind()
                           at System.Web.UI.Control.DataBindChildren()
                           at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
                           at System.Web.UI.Control.DataBind()
                           at System.Web.UI.Control.DataBindChildren()
                           at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
                           at System.Web.UI.Control.DataBind()
                           at Orion_Netflow_Summary.OnLoad(EventArgs e)
                           at System.Web.UI.Control.LoadRecursive()
                           at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

                        This has been reported to Solarwinds support anyway.

                        -Pete

                          • Re: NetFlow  v3.1 Upgrade
                            davidmaltby
                            PeteF,

                            FG4 stores only your uncompressed data.  If you reduce your 4 hour uncompressed down to 1 hour, and restart the service, then when the next night maintenance occurs and a shrink is performed on the database, then that disk will reclaim the space.

                            I've also looked up the details on your exception and have informed Tech Support about the details, so that they can help you further.

                            Thanks,

                            David

                        • Re: NetFlow  v3.1 Upgrade
                          Debbi

                          I did the 3.1 upgrade this AM (Tuesday) about 5:15 Pacific time.  I saw those 4 files get created but no change in their size since then.  I have not gotten any status events all day.  My swdebug.log file says the migration aborted right away at 5:30 but SW techies say the migration is still running.  I would like to see some evidence of that.  I feel like I am pretty much in the dark as far as what to expect.  My db is about 80 gig.  -Debbi

                            • Re: NetFlow  v3.1 Upgrade
                              davidmaltby

                              If you have a log message from the 'SolarWinds.Orion.NetFlow.Migration.MigrationDatabaseHelper' component that says that the migration was aborted, then the migration won't try to start again until you restart the serve.  If you could attach your swdebug.log file to this thread then, I can take a quick look and see if it gives me details as to why it aborted.

                              Thanks,

                              David

                                • Re: NetFlow  v3.1 Upgrade
                                  PeteF

                                  Hi David,

                                  I've responded to your email with the SWDEBUG.LOG. I can't find 'SolarWinds.Orion.NetFlow.Migration.MigrationDatabaseHelper'  in the swdebug.log.

                                   

                                  -Pete

                                    • Re: NetFlow  v3.1 Upgrade
                                      Debbi

                                      OK, here's a "let the upgrader beware" story.  My db is about 80 gig.  The migration kept aborting because the queries were timing out, but tech support thought it was still running even though it said clear as day in the log that it had aborted!  Then I had to set the SQL query timeout to "unlimited".  The db/log fiiles started growing by tens of gigs, and I thought I would run out of disk space.  I actually stopped netflow export overnight last night so the migration might finish up, but as of this AM it was still only 1/3 done, and I saw huge gaps in Orion NPM data overnight so this AM I had to stop Netflow (and thus the migration) completely and send the diag file to SW.  No response yet.  At the start, just after 3.1 was released, the netflow dev person was *out of the country*.  How can that happen?  There's only one person???  I had to constantly monitor the disk space and swdebug.log because the migration kept aborting for different reasons (had to restart Netflow).  So now I am left with no Netflow running, a SQL db with only "Simple" recovery set (SW recommendation) to reduce the log file size during the migration (cannot restore now if problem occurs except from last full backup), NPM database missing most of last night's data, and SW techies who are so busy with other cases they cannot consistently help.  No one seems to be able to estimate how much more disk space I will need if I continue the migration and even if I schedule it over the weekend, I will lose Orion data during the process.  I hope in the end maybe it will be worth it?

                                        • Re: NetFlow  v3.1 Upgrade
                                          Andy McBride

                                          Hi Debbi,

                                          Thanks for the feedback. I know this doesn't help you, but it looks like the upgrade issue you have experienced is not common.

                                          Our dev staff do travel to and from various SolarWinds sites but this usually only means that their hours of availability will be off sosomewhat from N. America support hours. Sorry if this was an issue. Orion data loss during migration should not be happening. I'll check with support on where this stands.

                                          Andy

                                            • Re: NetFlow  v3.1 Upgrade
                                              Debbi

                                              Here is the email I sent this AM to support.  I am getting pretty frustrated.

                                              (Case #79192)I am wondering if we schedule to backup the transaction logs every 15 minutes during the migration if the logs will not grow so huge.  I believe when they are backed up they shrink almost to nothing.  Would you check with your SQL folks about this idea?
                                               
                                              I have another case number now (80602) for data gaps in Orion NPM data during netflow migration.  I am not sure who is dealing with that issue or if you have taken it on as a part of this ticket.  So my issues are:
                                               
                                              1. How large is the database (.mdf) files expected to grow based on the previous numbers I sent you and where the migration stopped last:
                                              .mdf 81,220,736 KB
                                              FG1 8,692,928 KB
                                              FG2 1,787,072 KB
                                              FG3 17,093,376 KB
                                              FG4 751,872 KB
                                              2009-02-11 07:00:19,909 [7] INFO  SolarWinds.Orion.NetFlow.Migration.MigrationDatabaseHelper - Progress within the NTA30_NetFlowSummary2 table : 25.10% of data migrated. Overall progress 34.72%. RowsProcessed = 172441. Duration = 106.6752744 seconds.
                                               
                                              2. Will backing up the TAs every 15 minutes help mitigate the log file (.ldf) growing so huge?
                                              3. How can I fire up netflow/migration again without causing data gaps in Orion NPM?  ( I sent the diagnostics as were requested)
                                               
                                               I attach a .jpg of the data gaps we experienced before turning the migration off.
                                               
                                              I will expect a reply this morning.  If you cannot provide one, will you please give me the phone number of someone who has the power to get some answers for me?  I would like to run the migration starting tomorrow night and need the answers so I can do that.  This case has been open for almost two weeks.
                                               
                                              Debbi
                                                • Re: NetFlow  v3.1 Upgrade
                                                  Debbi

                                                  We have now given up on the migration.  The last error was at 40% when it  had a "serious error" when preparing to shrink the db and aborted.  Now they are telling me to truncate the old tables and let the migration complete, but I bet I will just get another error.  We are probably going to wipe out all of Netflow data and tables, uninstall, and start over.   I cannot spend any more hours waiting for support to get back to me.  I email them in the AM, I don't hear anything by afternoon, then I end up calling them.  I wish I were the people they were working with while I am waiting for them to work with me. :)

                                                    • Re: NetFlow  v3.1 Upgrade
                                                      davidmaltby

                                                      If you're available tomorrow morning, I'll get on a GoTo meeting with the Tech Support rep and yourself to resolve this.

                                                      Sorry about all this trouble that you're having.  I have not seen any other cases where a customer's migration aborts like this.

                                                      Thanks,

                                                      David

                                                        • Re: NetFlow  v3.1 Upgrade
                                                          Debbi

                                                          Thanks, David.  I took yours and the tech's advice and instead of starting from scratch I dropped the 3.0 tables (I did not need the unmigrated data) and did a db shrink.  It completed in a few minutes on a db of over 108gig but I only gained back 6 gig even though one of the 3.0 tables was 33 gig.  I thought that was strange.  I will see what happens after the nightly maintenance.  I have restarted netflow and the migration did not start, as expected (since I had deleted the old tables).   I will keep an eye on it and report back to the tech tomorrow AM.

                                                          I would like to have an understanding of what goes on in the swdebug log during normal netflow use.  What does all that stuff mean?

                                                          -Debbi