11 Replies Latest reply on Jul 9, 2012 5:36 AM by Deltona

    Is administering storage hard for you?

    Fabio Rapposelli

      Hi Thwack, I'm Fabio Rapposelli, one of the thwack Ambassadors for the month of July, I'm a consultant on everything related to Storage and Virtualization, VMware Certified Design Expert #58, have a soft spot for storage startups and I'm also a recovering Networking guy :-).

       

      In my years as a storage guy I've been exposed to some of the most eclectic way to keep track of your storage utilization and configuration: Text files, Excel spreadsheets, text-based Databases, Access files, home-grown full-fledged web applications and, last but not least, an RPG program running on an ancient AS/400 (!!!).

       

      Sometimes, when the storage estate is particularly large, these methods are also used to perform chargeback activities both internally (corporate-wise) or to customers (in an SP scenario), not to mention provisioning tasks and data protection, that sometimes rely on configuration files generated by the before mentioned spreadsheets and databases.

       

      The common theme that brings all these approaches together is that storage is hard to administer, what is your way to tackle this problem? Do you agree that storage is hard to keep track of? Did this thing ever bother you (or your boss) so much that it became the major driver to move on to another storage brand?

        • Re: Is administering storage hard for you?
          familyofcrowes

          We recently had a power outage in our datacenter.  As we were trying to recover the SAN was not responding.  Managers started asking were the documentation is for the SAN, meaning what data is where, what servers use it for what drives and how immportant is it.  Well, all that information is on the SAN, in spreadsheets....  for over 500 servers....

          This information is something that many just feel "will allways be there".  Noone thinks the SAN is die.... or even have any issues...  But it does....

          I can say that something that is automatic and NOT ON THE SAN is a good idea.  We have 4 generators and many other backups, but still, the whole datacenter went black for 3 seconds....  and the SAN DID NOT LIKE THAT....  36 hours later they had everything back o normal, but with good documentation that was quickly available would have drasticly reduced that time....

           

          Time to see what Storage Manager can provide...

            • Re: Is administering storage hard for you?
              Fabio Rapposelli

              familyofcrowes wrote:

               

              I can say that something that is automatic and NOT ON THE SAN is a good idea.  We have 4 generators and many other backups, but still, the whole datacenter went black for 3 seconds....  and the SAN DID NOT LIKE THAT....  36 hours later they had everything back o normal, but with good documentation that was quickly available would have drasticly reduced that time....

               

              This is definitely a golden rule, even if the Storage and the SAN are redundant they can fail, my opinion is that you should always keep your storage, SAN and (especially) your DR documentation safe in a secondary location, and, if applicable, in paper form.

            • Re: Is administering storage hard for you?
              tomaddox

              Short answer: yes, but it's gotten easier. It used to be that you simply didn't have the luxury of making a mistake: arrays, controllers, and servers were all deeply unforgiving of configuration errors, and that issue has certainly improved over the past five years. On the flip side, there are still ample opportunities to make mistakes. Especially when connecting via Fibre Channel, the dependency list is both long and demanding. One must make sure the array is running the right storage code, that the HBAs have the right firmware, that the multi-pathing software is the right version, that the drivers are the exact right revision, etc., to say nothing of zoning, initiator registration, ALUA configuration, the list goes on and on, and that's just to get a host to see the storage in the first place, never mind troubleshooting performance. Once you get down to the performance level, there's a whole additional host of factors to consider, including RAID type, number of spindles, cache utilization, block size, data classification, etc., so that the knowledgeable storage admin actually has to be quite aware of what applications are running atop the storage infrastructure.

               

              The rise of flash storage has taken some of the pressure off, but more complexity has arisen around storage tiering vs. caching, and cost/IOPS (or throughput) vs. cost/GB. A view of the big picture is now essential--it's no longer enough to throw more disk at a storage problem until it goes away (although this is still an option, it makes one look rather behind the times). Unfortunately, one is faced with the choice of an unproven vendor with an impressive-sounding solution or a proven vendor with technology that merely evolves existing paradigms instead of radically revamping them.

               

              Worst of all, when it finally comes time to migrate to a new storage platform, you will find that older servers are not certified against the new platform and that there's no clear migration path, essentially requiring that legacy storage be kept around until the older servers can be retired, or forcing an upgrade to servers that might not necessarily need it.

               

              Not at all helping the problem are the big storage vendors themselves. Facing intense competition among themselves and from young and hungry startups, they do everything the can to spread fear, uncertainty, and doubt, obfuscate true performance and utilization numbers, refuse to cross-certify storage virtualization platforms, and just generally make it that much harder to choose the right product for one's needs. There is also relatively little thought given to seeing precisely what loads are consuming which resources or creating performance bottlenecks, so pulling out the needed information can be quite an ordeal.

               

              On top of all these other issues is the fact that storage remains surprisingly expensive. It can be very difficult to explain to end users, who know that they can pick up a 2 TB hard drive for under $200, that the price of 2 TB of fast, RAID-protected, enterprise storage can be a hundred times the cost.

               

              All of these factors are no doubt a big part of what's driving enterprises to cloud-based solutions. Storage virtualization allows companies to make more efficient use of their storage investment (or to offload that investment entirely to an external provider) by more intelligently allocating capacity and performance based on policy requirements rather than strict resource assignments. Of course, the people who design those systems still need to know what they're doing, and sufficient resources need to be available, so there's no escaping the fundamental need to monitor and manage the storage layer.

               

              In short, storage remains a challenging resource to manage, requiring a non-trivial amount of specialist knowledge. Even tools which purport to simplify the process of storage management are so complex in themselves that setting them up and navigating them can be quite the chore.

                • Re: Is administering storage hard for you?
                  Fabio Rapposelli

                  tomaddox wrote:

                   

                  In short, storage remains a challenging resource to manage, requiring a non-trivial amount of specialist knowledge. Even tools which purport to simplify the process of storage management are so complex in themselves that setting them up and navigating them can be quite the chore.

                   

                  I think you really nailed the point, undoubtedly, storage is hard to manage, softwares can help, big time, but what I see as the highest hurdle in them is that you need to adapt your mindset and work as they were intended.

                • Re: Is administering storage hard for you?
                  Deltona

                  Hi Fabio

                   

                  Storage IS hard to administer imo, harder still for those that don't understand the fundamentals behind it. The storage market is a huge cow that is being milked by the vendores because of that (them). The high price for enterprise storage comes in on the storage management/software layer. Every vendore has their own software and that is in essence what you're paying big bucks for. The chassis, disks, raid controllers, batteries and interconnects cost *nothing* to produce. The software however, its ability, feature set and as tommadox mentioned "impressive-sounding solution" is what attracts those who don't understand fundamentals. Most of the time it is the software that makes storage administration difficult.

                   

                  Common storage technology for enterprises hasn't changed *that* much in recent times, its still the same hardware, and most of the times and in 80% of cases, the hardware is outsourced and rebranded. Of course, vendors won't disclose those details. The changes or additions rather, are made to accomodate the vendors "new" software features that would - in practice - aid or boost performance. Again, this doesn't help make administration easier, added complexity rarely does.

                   

                  Something that would make my life easier would be an SRM system with a built in IOPs calculator which you could use to calculate IOPs and pin the results against your monitored storage groups. That way i wouldn't have to use a calculator and macro spreadsheet everytime i need to calculate theoretical and practical IOPs available for my systems. I'd also be able to calculate congestion and address a whole set of other issues that may/will arise more efficiently.

                    • Re: Is administering storage hard for you?
                      Fabio Rapposelli

                      Deltona wrote:

                       

                      Something that would make my life easier would be an SRM system with a built in IOPs calculator which you could use to calculate IOPs and pin the results against your monitored storage groups. That way i wouldn't have to use a calculator and macro spreadsheet everytime i need to calculate theoretical and practical IOPs available for my systems. I'd also be able to calculate congestion and address a whole set of other issues that may/will arise more efficiently.


                      That would be the holy grail (and would probably put me out of business ), unfortunately estimating IOPs is akin to black magic, there is no (to my knowledge) product in the market right now that can do this, but you could reach the same goal aggregating data from different equipments like SAN probes (to get congestion data and pinpoint queue depth issues), Storage metrics (IOPs, R/W ratio, avg. block size, cache utilization) and HDD spec sheets (avg. seek time and others).

                       

                      But that would still imply a monster spreadsheet

                        • Re: Is administering storage hard for you?
                          Deltona

                          Well you could start by keeping it simple to a disk level and combine that data with real-life test though benchmarking.

                          This is what i used to do when i first figured out how to calculate IOPS.

                           

                          Get rotational speed, average latency, average seek times.

                          Multiply or divide accordingly based on storage set/pool type, RAID etc.

                           

                          Then run benchmarks against the storage set, with and without load. This usually took up to two weeks.

                          Finally compare scenarios, measure theoretical vs practical gathered data and optionally, calculate an average.

                           

                          Running benchmarks against the storage sets isn't something an SRM system would be able to do for you but then again, benchmarking is a must before goig live.

                          The calculation on the other hand is definitely something that the SRM would be able to manage...

                      • Re: Is administering storage hard for you?
                        cmgurley

                        Since most of the questions have been adequately addressed (in a way with which I agree), I'll skip to the last one:

                         

                        Did this thing ever bother you (or your boss) so much that it became the major driver to move on to another storage brand?

                         

                        The answer to this question changed my position on whether storage management is hard (it isn't now). We started our SAN life on EMC CLARiiONs and lived decently well for a couple years. However, at least once a year, I'd have to pull out my spreadsheets and whiteboards to rearchitect the LUNs and RAID Groups to address growing needs. Then came the fun of shuffling the data, where possible, hot. Granted, our storage management has always been about the easiest I can imagine because we're a 100% VMware shop, running production, mission-critical apps on it since 2006. Even so, a month or more or data shuffling due to blocky architecture was cause enough for us to jump vendors when the next refresh came around.

                         

                        For us, that move was to 3PAR (now HP 3PAR). We moved our production over in '08 and the rest in '11, and it's been nice--hard to easy. The storage architecture is fluid and 3PAR's abstraction freed us from the RAID Group handcuffs we'd been bound to for two and five years, respectively. That said, even 3PAR isn't without caveats and I've blogged on a few (www.bctechnet.com), but it's served us well.

                         

                        The overall hard/easy question for us is best addressed by virtualization. Managing connections, zoning, HBAs, etc for a few big hosts is SO MUCH nicer than doing it for the hundreds we would have if we were fully physical. The saying used to go, "Ignorance is bliss", but now I'd say that abstraction is.

                         

                        --Chris

                          • Re: Is administering storage hard for you?
                            Fabio Rapposelli

                            cmgurley wrote:

                             

                            For us, that move was to 3PAR (now HP 3PAR). We moved our production over in '08 and the rest in '11, and it's been nice--hard to easy. The storage architecture is fluid and 3PAR's abstraction freed us from the RAID Group handcuffs we'd been bound to for two and five years, respectively. That said, even 3PAR isn't without caveats and I've blogged on a few (www.bctechnet.com), but it's served us well.

                             

                            I love to see ease of administration and dynamic data management as major driver to change a storage platform, sometimes the only thing decision makers care about is price. :-)

                          • Re: Is administering storage hard for you?
                            byrona

                            Managing storage is definitely a challenge for us and I am not sure there is anything you can do to make it easy, maybe just slightly less difficult.  We currently exclusively use NetApp storage systems and the tools that NetApp provides seem to provide a log of insight into performance once you take the time to learn them.  My major complaint about the tools is how many different ones that they have, I would prefer it to be consolidated into one tool.  We also have Orion that alerts when the status is in any form of degraded which will trigger us to dig deeper using the NetApp tools.

                             

                            I find the CLI management of the NetApp to be very old and clunky feeling.  I think having a CLI interface is important; however, I think a GUI is preferable for the 90% day-to-day management tasks due to the visualization helping you to understand what you are doing and I think NetApp is lacking here as well.

                             

                            The other challenge seems to be tracking what storage is being used where.  The newer NetApp tools seem to provide some of this functionality but I haven't spent a lot of time with them yet.

                             

                            I would imagine storage management would be much more difficult in an environment where you have multiple storage vendors, I fear at some point we will face that challenge and that's when we may need to consider the SolarWinds storage management solution.

                              • Re: Is administering storage hard for you?
                                Fabio Rapposelli

                                byrona wrote:

                                 

                                I find the CLI management of the NetApp to be very old and clunky feeling.  I think having a CLI interface is important; however, I think a GUI is preferable for the 90% day-to-day management tasks due to the visualization helping you to understand what you are doing and I think NetApp is lacking here as well.

                                 

                                Management fragmentation is a real plague for NetApp, I completely agree with you that CLI is important, and NetApp's is comprehensive, documented and can go pretty deep inside the filer's internals, but I can't really say the same thing for the GUIs, fragmented and sometimes too "high-level".

                                 

                                I saw Solarwinds Storage Manager implemented in a pretty large NetApp farm and was quite impressed, make sure to give it a spin and compare it to the new OnCommand from NetApp.