cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

One Company's Journey Out of Darkness: Part I - What tools do we have?

Level 10

I've had the opportunity over the past couple of years to work with a large customer of mine on a refresh of their entire infrastructure. Network management tools were one of the last pieces to be addressed as emphasis had been on legacy hardware first and the direction for management tools had not been established. This mini-series will highlight this company's journey and the problems solved, insights gained, as well as unresolved issues that still need addressing in the future. Hopefully this will help other companies or individuals going through the process. Topics will include discovery around types of tools, how they are being used, who uses them and for what purpose, their fit within the organization, and lastly what more they leave to be desired.

Blog Series

One Company's Journey Out of Darkness, Part I: What Tools Do We Have?

One Company's Journey Out of Darkness, Part II: What Tools Should We Have?

One Company's Journey Out of Darkness, Part III: Justification of the Tools

One Company's Journey Out of Darkness, Part IV: Who Should Use the Tools?

One Company's Journey Out of Darkness, Part V: Seeing the Light

One Company's Journey Out of Darkness, Part VI: Looking Forward

Lean IT teams often do whatever they can to get by and my customer was no exception. One of the biggest challenges they had in approaching their network management strategy was to understand what they currently had. We had to work through the "day in the life" of a number of individuals to identify the core tools used, but were constantly surprised by new tools that would appear or were used so infrequently that the team would forget about them until a specific use case arose.

Open Source Tools

The team found open source tools to be of tremendous use, especially Netman and MRTG.  These provided much needed visibility and the price was right given the lack of investment in monitoring tools at the time of deployment. The relatively complex nature of deployment of these tools did limit adoption and we found that often these tools lagged behind from a configuration standpoint.  New equipment would be deployed without necessarily being integrated into the tool, likewise old equipment when replaced was not always removed from the tools. Lack of policy and discipline in a busy IT shop had effectively limited the effectiveness of these tools.  This was further compounded by only a small subset of the team having access. Additionally, as an outside resource, I had no idea what "normal" was when looking at the tool (e.g. is that router down or has it been removed?).


Vendor Specific Tools

These tools are something most are familiar with products like Cisco's Prime Infrastructure, Aruba's Airwave, VMWare's vSphere Operations Management (VSOM), etc. Each of these tools had been deployed widely and would tend to be used by those who's job responsibility primarily covered the area managed by the tool, however others that could benefit from this tool very rarely used it if at all. These tools tend to be fairly expensive and offer many features that are typically not leveraged very well. Additionally, most of the tools have robust AAA capabilities that would enable them to be shared with help desk teams, etc. but these features had been overlooked by the team, despite having been properly configured for their own purposes.


Third Party Tools

Some investment had been made in third party tools, typically around a specific need. A good example of this would be the Kiwi Cat Tools for ease of device backups. While this functionality existed in other tools, the company wanted a single location for all device configuration files. The customer found that numerous tools existed, but it took the entire team to enumerate them and in a couple cases multiple instances of the same tool had been purchased for usage by different teams.


Scripting

Certain members of the IT team who were comfortable with writing and using scripts would develop their own tool sets, however these would often not be shared with the rest of the IT team until some specific project jogged the author's memory who would then offer up some script that had been written. In all cases these were very specific and had never been fully socialized, the team decided to develop a website internally to reference these tools and their use cases.


Taking a Step Back

Working with each of the administrators and their areas of responsibility it was easy to understand how they've gotten to this point where substantial investment had been made in a myriad of tools without putting a strategy in place. Each of the teams had acquired or deployed tools to make their lives easier and tended to go with whatever was vendor aligned or free. Taking a step back together from it all and looking at the system in its entirety provided a much different perspective - is this really how we'd design our management infrastructure if we built it from the ground up? Clearly not, so what next? Looking at the tools currently deployed, it was obvious that substantial duplicate functionality existed as well as a number of gaps, especially as it pertained to any one specific team's visibility.

Enumerating the existing tools, processes and use cases highlighted how much organizations actually do spend on tools while complaining that they don't have the visibility needed. Open lines of communication between teams, the development of an official or virtual "tools team", and careful consideration of products purchased are key to the success of running the IT team properly. Highly custom scripts and those who can write them can be of great value to an organization, however this value is wasted if the team at large isn't aware of these scripts and how to best utilize them.

24 Comments
MVP
MVP

As you mentioned open lines of communication and documented procedures for adding/removing resources go a long way to keeping things up to date.

I understand the growth issues and every team wanting to monitor their own world....I have never seen that work well for a large company unless there is a set of guidelines such that

the notifications come through a single source for consistency. 

Good write up...

Level 10

Thanks for the feedback! A colleague of mine who came from the operations world used to be a huge advocate of a dedicated tools team. This team whether dedicated resources or a virtual team would be responsible for the deployment, configuration, and enabling of other teams within IT. There are so many uses cases in which IT tools could provide far better visibility to an organization if deployed to additional user groups.

I'm advocating for a more global adoption of a more universal tool platform (Orion) across our multiple silos.  There's push back by users who feel the freeware version of Nagios is the right price point, or even the paid version ($8K) better suits their comfort levels for budgeting.

I can sympathize--Orion isn't free, or cheap.

But from what I've seen, if you're going to go professional, you should be using professional quality tools.  Those tools should provide interfaces and reports that look good, have options for multiple types of displays, show color in graphs and charts.  And they should be well supported and intuitive.

Orion's been that for me.  My challenge is to get that "Orion foot" into their silos long enough to get it configured and displaying what it can do so well.  Then finding folks in those departments who can appreciate it, and will begin using it.

Sure, change can be unpleasant.  But is doesn't always have to be.  I found changing from MRTG and CiscoWorks to NPM and NCM to be a very positive experience.

Level 21

rschroeder‌ I have been in that exact situation and occasionally have to have that same "Nagios" discussion with somebody.  Some people think that because they don't have to pay for it they are getting a real deal; that couldn't be further from the truth in many situations.

Level 21

This article provides a great use case of the different tool types that end up in so many environments.  Ultimately having a level of oversight of the entire toolset of a company is important to avoid the overlap and maximize the value.  We are actually entering a new phase of our company where we need to re-evaluate our entire tool base and potentially  re-organize it all for the new direction of the company and it's a huge task and it's not at all easy to cover all the bases, minimize overlap and maximize the value.

And then I realized I've seen this before . . .

Too Many Tools

Level 10

Definitely hope you feel differently as we go down this path. I think the plethora of tools is a common theme so I wanted to make sure this follows a different course. I did feel the need to start with the discussion though so that I can share why we did what we did. I'm not saying its the right way or the only way, but rather one way that worked for a large organization.

Level 10

Ease of use of these tools is definitely a consideration.  Nagios isn't the easiest thing to get up and running and does take quite a bit of care and feeding.  Good Nagios deployments are pretty impressive though.

Level 10

Have you guys discussed the concept of a virtual tools team? looking at rschroeder‌'s post talking about "their silos", a tools team can create a sense of community that can aide the deployment of tools across a wider base. A lot of my work is in healthcare and those teams tend to be very siloed. I've shared ideas of getting a network monitoring tool rolled out to biomeds for instance and often the network team had never even thought of that before, despite major benefits of providing that visibility.

The comment is a happy realization that this is an ongoing concern, and noticeable progress is being made.  I like the two threads, and find good value in both, and in seeing the differences a year makes.

I'm highly in favor of this, and am working towards it in my organization.  There are tech-oriented teams that are glad to learn there's something another team has bought, which others can leverage.

And there are technophobes who are pleasantly amazed that something intuitive and user-friendly is available to them, that will help their team move forward, without any additional investment in contracts, licenses, or hardware.

Our health care teams that I'm trying to bring Orion to include:

  • Biomed
    • Hundreds or thousands of wireless pumps are present in our nearly 100 hospitals and clinics, but sometimes some health care staff hoard them. 
    • Biomed and Management don't want to have to buy more if there are many going unused, hidden in janitors' closets for local ward use. 
    • HEAT Maps show them all so they can be redistributed.  HEAT maps also help Biomed get these devices upgraded and maintained without wasting a lot of time searching for them
  • Security
    • Wireless interference sources like Blue Tooth, 2.4 Mhz wireless phones, X-Boxes, wireless head sets, and personal cell phones with WiFi hot spot functionality enabled impact Vocera calling systems. 
    • Finding those Vocera badges during a security event is made a lot simpler with HEAT maps
    • Reducing interference via the HEAT maps makes Security's job easier, and makes the staff safer when they can contact Security via Vocera quickly and accurately
  • Maintenance
    • One of the main reasons for RTLS is finding equipment.
    • This extends to wheel chairs that are not returned to places where they're needed.  In both cases, HEAT maps are a new tool for many departments to use.  Some folks never have heard of this technology
  • Staffing
    • Leaving the Big Brother implications behind, when a patient or doctor needs a specialist fast, if they aren't using Vocera then they could be out of touch with that specialist.  HEAT maps and RTLS chips on ID badges help here a lot.
  • SAN support
  • SAM users
  • Web Monitoring for Applications staff
  • Database admins
  • Telecom doesn't have a view into the network, and VoIP and Video Conferencing is only getting more wide spread in our organization.  When I bring the tools to Telecom I expect suspicion, then incredulity, and then smiles and maybe even laughter or a happy tear when they realize what my Team can make available to them with Orion modules


Well, you can go right down the list of SW products and create a list of specialized teams that are on my list to introduce to Orion.  This will be a happy and friend-making year . . .

Level 14

Good write up.  NPM has helped us move forward as far as communications between teams.  One member of our network admin team acts as the "salesman" for the other teams so we can all be on one page.

Level 10

Thanks! By salesman he's out sharing the virtues of the platform and getting buy in from the other teams? Seems a reasonable way to go about it.

Level 14

Yes.  We hold regular training classes with our teams and upper management of other teams to demonstrate the interface and ease of use.

That's a goal for my organization to shoot for:  " . . . hold regular training classes with our teams and upper management of other teams to demonstrate the interface and ease of use."

I like it!

MVP
MVP

Something else that really helps...especially with the teams who are your customers...hold weekly or bi-weekly meetings to talk about topics such as pain points, changes in the near future, what-ifs, how can we help you to do your job better, how can you help us to make your job easier, problems looking for a solution, new solutions looking for a problem, etc.

This fosters an environment of communication and helps you to stay aligned with the other groups.  Tool topics are appropriate so that all parties are aware and can leverage the best of the tools in the tool box.

We are in this process now of asking that question.  For years our team has, fought to unify the tools and get a single pane of glass.

Our new director, recently took a list to our CIO with all of the overlapping tools.  She stopped county at 60. 

The thing we fight. 

Everyone trusts THEIR vendor

Everyone wants to OWN the tool

Everyone wants to MAKE the rules


and as a result the network is flooded with triplicate traffic.

Level 10

Part two of this series is now up. Thank you all for participation thus far, please let me know what you think!

MVP
MVP

Thank you !

MVP
MVP

Regarding the Open Source Tools section...I thought CPU monitoring on ASRs was convoluted in SolarWinds.  It is, but somehow in Cacti was even more painful, which I didn't think was possible.

Level 10

We've had a similar issue with the battle over tools and shadow IT. One of the things that helped us was to create a share point site that lists all of the tools we currently use as well as their current version and a URL to access any web GUI it uses. We also instituted a weekly change management cycle that any changes being made to a production system have to be approved.

Level 10

At my previous job we had plenty of open-source tools as well as some cheaper tools. Always had to cut costs.

MVP
MVP

From a network point of view, we mainly use Solarwinds for all our networking needs. It also helps other department seeing what's going on on the network as it's easy to use and read information on. Even though Solarwinds costs money, it is time efficient.

Level 13

Reading through old posts obviously.  Man, this is so where we've been.  MRTG, custom RRdtools (me being familiar with scripting), ORCA for our Solaris boxes, Nagios, Cacti, etc.  Now we're on Orion and have nearly consolidated everything we had that was custom.  So much better and you only have to learn one tool.

About the Author
Shaun Neal is a Solution Architect with enterprise networking, security and mobility expertise. Additionally, Shaun is engaged in wireless product development, deployment, integration and go to market strategies. His experience aligns information technology and the organizational mission to create service orientated architecture design and see it through implementation.