cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

How Well Do You Know Your Environment?

Level 11

Do you know what's in your data center? How about your wide area network (WAN)? If you had to draw a map or produce a list of all the things that are connected to your systems in the next week, could you? It sounds like the simplest of things to have, but more often than not, most people have no idea what's really going on in their IT organization.

Years ago our VAR took on a new client. This client was in the medical field and had a really good idea of the technology in their organization. They knew everything that supported their mission to provide value to their customers. However, the senior engineer from our company that was supporting the client wanted to map the entire infrastructure before we took them on. The client told him that it wasn't necessary. He insisted. He spent weeks mapping out every connection. He looked at every device and traced every cable. He produced a beautiful Visio drawing that ended up hanging in their office for years like a work of art.

What did our senior engineer find out? Well, as it turns out, one big thing he found was a redundant wireless bridge on the roof that was used in the past to connect to a building across the street. When he first discovered it, no one knew what it was supposed to do. It took a few days of questions before he found someone that even remembered the time when the company rented space from the ancillary building and wanted it connected. When we brought up the old equipment to the client's IT team, you can imagine the quizzical looks on people's faces. Well, except for the security team. They were more worried than curious.

Why is it so hard to keep track of things? How is it that rogue equipment can appear in our organization before we realize what's going on? In part, it's because of the mentality that we've had for so long that things need to "just work." Instead of creating port security profiles and alerting people when someone plugs a device into the network, we instead choose to enable everything in case someone moves a computer or needs an additional drop activated. Instead of treating our user space as a hostile environment, we open it up in the hopes that our users don't call us for little things that need to be dealt with. This leads to us finding all kinds of fun things plugged into the network causing havoc by the end of the day.

Likewise, we also don't have a good plan for adding equipment behind the scenes. How many times has a vendor offered a proof-of-concept (PoC) trial of equipment and plugged it directly into the network? I'm sure that some of you out there with an Infosec background are probably turning colors right now, but I've seen it more times than I care to count. Rather than taking the time to test equipment with good testing data, the vendor would rather test the equipment against live workloads and push traffic through a PoC to show everyone what it really looks like or how easy their equipment really is to work with.

If you don't know what you're working with in your IT environment, you might as well be trying to work with a blindfold on. You may have switches running as the root of a spanning tree the are from the last century. You may have older virtualized hosts that aren't getting patched any more. You may even find that someone has installed nefarious hardware or software to collect data without your knowledge. And all of that pales in comparison to what might happen if you work in a regulated environment and find out someone has been quietly exfiltrating data around a firewall because you don't have proper controls in place to prevent it.

How well do you know your IT organization? Do you know it well enough to point out every blinking light? If you had to disappear tomorrow would your co-workers know it as well as you? Do you document like your replacement will come looking for you when things go wrong? Leave a comment below and let everyone know how well you know your world.

7 Comments
Level 14

I know my systems.  I have to as no one else cares.  I was contracting for 20 years and would move jobs about once a year.  Every time I moved I would have to get to know the new setup and the best way to do that was to map it out.  It's now second nature and has frequently really helped.  I've lost count of the number of times I have been in a meeting where they were planning some changes and I have stopped them as they were going very wrong because of assumptions about the network and systems.  I recently caught a senior techie trying to place virtual system management servers on a publicly routable vlan with no firewall.  Oops.

I am glad to say that while I can relate to every problem mentioned, we have taken strong steps to make all of them better. Audits and automated discoveries get compared to a CMDB. Ownership and support details get reviewed.  When a flaw in the process is found, we address it.

In a large organization it can take a lot of effort to stay on top of the documentation, but when you need it, nothing replaces it.

It was years ago now, but we had an internet connection that we knew was there and nobody knew how or why it got there. We knew that certain web services were donated to us by virtue of our traffic using their network. We knew it was dark fiber that we and the donor lit up. No one knew how it started. No one knew how to fix it when it died. And one day we had the fiber cut. It took a long time to get people to acknowlege what it was, who could do something, and how the fix would be done and paid for.

Today if that happened, it would be much quicker, and much less awkward. Both endpoints and the owner of the fiber have everything they need documented, and there are no more mysteries. When it comes to problem solving, surprises are rarely good.

MVP
MVP

Nice write up

Level 13

Good Article. Local Knowledge is good but documentation is King. The Users won't wait for you to get back from A/L to fix their problem when something goes wrong.

Level 20

Rogue devices are a big deal for us... wireless and wired.  It's becoming a part of daily operations and part of the continuous monitoring state desired by RMF - Risk Management Framework.

Level 16

NAC ?

ForeScout will fix that ...

www.forescout.com

Level 15

After 34 years in the IT field and most of it spent in Field Service, the one resounding fact that the first visit to a new customer was to map everything out.  Some clients were confused that I was not just fixing the issue with the servers or endpoints, but when I explained that I could just single in on the pain point or take a few extra hours (not always billable) and produce a large, big picture view of their environment.  I would then provide that to them as well as store in our shop.  That way, any time a call came in, our techs could bring up the diagram and understand where and what was affected.  It allowed us to be much more proactive in resolving issues for our clients.  Strange though, in my current position, the system administrators are siloed in their systems and don't have or don't want to share how the systems interact.  I inherited detailed drawings on the network (kudos to a prior network administrator) and I have continued to keep them updated.  We have policy that keeps unconnected ports from remaining plugged in and when moves occur, the unused wires are unpatched.

Great article!

About the Author
A nerd that happens to live and breathe networking of all kinds. Also known to dip into voice, security, wireless, and servers from time to time. Warning - snark abounds.