cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

IT security tales from the crypt…err data center

Level 16

I was working in the data center, late one night

When my eyes beheld an eerie sight

For my SIEM began to screech

And suddenly there was a breach…

Ok, ok—that was a bit cheesy. But hey, it’s October, which means two things:

  1. It’s Cyber Security Awareness Month
  2. It’s almost Halloween

So how could we not take advantage of the opportunity for a little SolarWinds-style “Monster Mash”? You know, one thing security incidents have in common with the “Monster Mash” is that they both certainly catch on in a flash (more top-grade cheddar!).

But seriously, what better time of year than spooky ‘ol October to focus on cyber security? After all, security incidents have a tendency to create real-life nightmares. And so, in honor of both Cyber Security Awareness Month and Halloween, we’d like to hear your craziest, scariest security tales from the crypt…err data center.

Using the comments section below, please share your spookiest IT security stories—big or small, truly scary or a little funny—and how you were able to get out of the nightmare alive, or at least what lessons you learned.

Share your thoughts with us by October 16, and we’ll give you 250 thwack points (sorry, no candy).

38 Comments
Level 14

A few years back (2002-2003) I was working for a large financial institution, managing a systems and networking group. It's 7:20 AM and as I walk into my office I overhear 3-4 of our system security people talking in an animated fashion about something. Went over to say good morning and I hear that they are talking about a virus that appears to be ping ponging all over the building and sending emails to all of our contacts externally. It had been active for about 30 minutes or so. They told me that it was being spread by an email that was supposedly "penned" by a senior person at the company... Their discussion was all about the virus and the type it was and why the virus software had not caught it.... and so on and so on...

So I asked.... "So we are in the process of taking the email servers off the network and isolating the problem and stopping the threat from spreading .....  right?" The sound of crickets permeated the area..... Picked up my Nextel and I called one of my guys... 3 minutes later... issue contained...(300 PCs affected out of 2800).... 

Five hours later... resolved and back to normal.  I related this to their manager.... a bit later in the day..... His face had that look of horror that only a classic OMG moment can give... (Needless to say there were a lot of sad faces about 45 minutes later after his team meeting)..

Never let common sense get in the way of a good discussion.....

It was a week after 9/11 and we got hit with a virus. Nimda. First we didn't know much about it, was it part of the attacks from a week before? What was it trying to do? It didn't take long to slow and stop it, but we still had manual clean up to do. I got assigned a trip to clean the systems that were impacted at the airport, the company I was working for had a few private jets in a hanger there. The airport was shut down, they faxed a copy of my drivers license ahead of me so that I could get through the checkpoint that had been set up just outside of the airport. The entire time I was there it was a ghost town, only a few support staff were in the building. Every few minutes a National guard jet took off or landed from patrolling the boarder.

I was spooked.

Level 11

My data center is a converted patient care unit, located directly below the Labor and Delivery unit. Recently, the hospital remodeled the L&D Unit. Nearing completion, plumbers were installing the final touches and fixtures and the whole process went quite well. The two site-experienced plumbers were called off the site on the last day for another job and they sent in a new plumber to finish. The new guy didn't know the area he was working on worked off a different water main than the area finished already and uncapped a pipe plug that was charged to 125psi. A geyser erupted and after four inches of standing water in the unit, it finally overflowed the new sealed flooring and found its way into the data center. So here we are with buckets, tarps, shop vacs, and plastic garbage bin bags trying to shelter all our equipment. One of our critical systems racks was finally safe with the water barriers, so I opened it up to see what might have been damaged and on the very top system, I found someone had left a keyboard attached to one of the servers. It had captured all the water that leaked into the rack and kept it from pouring into one of the $50K servers directly underneath it. Talk about dumb luck! The security nightmare was all the people from IT and the construction crews running amok in the DC. We finally called campus security to post watch at the doorways to ensure nothing walked out and anyone who walked in was properly credentialed!

Level 10

I used to work for a datacenter colo facility. We had the usual badge in front door of the building, then biometric scan for our floor of the building. So we had this guy working for us that was a real brainless sort. We found out he was snagging the key for the fire exit and disabling the buzzer, then sneaking out to go for a smoke or to the bar next door or whatever; leaving the fire exit unlocked. I was a supervisor at the time and had to work 14 consecutive days over various shifts to cover firing him. Lucky for us, no one ever wandered in the back door.

Level 14

Great story....

Level 15

Well I had a lot of work in employment that was IT supervisor. The culture here n BR is still very poor for investment in the IT area.


Necessary was put broadband in the laboratory building, and this was more than 180 Mts of which had the router then according to the IEEE standards I should not spend a UTP cable for connectivity between buildings.


So as I had time to connect with what they're patterns, made 3 optical fiber budgets and sent to the direction sign ... in the course of the week I dreamed that the No. director had signed the authorization to purchase and the day next I was held responsible for not having resolved.

One day before the event I was told that the director had not received budget and there fore would be unable to resolve within the IEEE standards.I felt so much at that time, but I found my strength to think of a solution practice and UTP cables and so that worked smoothly ..
My IT room to the auditorium was about 200 meters, the managing director of the room mas were only 130 meters. Yet it was not appropriate, but that's what has to do. I spent a UTP cable to a router in auditorium and got free wi-fi access to the room, but that was the last time I did it. Here in Brazil the services that are we say 'gambiarra" kkkkkkkkkkkkkkkkkkkkkkkkkkkkkk


I got tired of working at that company. Everything was out of the standard, and that's too bad even for professional development.

That I still have other stories of when the primary server stopped kkkkkkkkkkkkkk but then I tell.






Level 14

Once again I date myself.

        This story goes back to my Windows NT 4.0 days.  It was at the twilight of my Naval career.  I was the lead system administrator for the unclassified network on my final ship.  We had a script that would parse through the proxy logs looking for obvious "bad" site keywords.  The output of the script would identify the username, machine name, and site visited.

        We had two different proxy server policies that would be used, "inport" and "underway".  When we were in port, we had a very wide pipe and everyone had access to the internet.  Underway, we were very limited on bandwidth and access was limited to people needing it for "official business" only.  Sailors who did not have this level of access were out of luck, unless of course they could get the username\password combination from a buddy who had official access.

        One Monday morning, I was reviewing the output from the script and discovered that a  particular machine was being used heavily for looking at "bad site".  The username was one of our NT Server system accounts!  This account password was held only by system administrators and was never used to actually log into any server.  It was allowed through the Proxy server only because it had system administrator privileges.

        I did a little digging into the machine by remotely logging into it.  It wasn't running NT, but was running Windows 2000 and was attached to the domain!  How was this possible?  A little more digging revealed that the user had loaded Cain and Able and had cracked the SAM from the Domain controllers.  He had all 5000 usernames and passwords for the network!

        After turning this information over to my chain of command, the computer was confiscated and the young man got to go have a conversation with the Commanding Officer.  Turns out it was his girlfriend who was surfing the "bad" sites, while he was there of course.

        This young man had the curiosity, tenacity, and technical ability to gain total access to our network and he was undone by his ego, showing off his "mad hacker skills" to girlfriend.  Secretly, part me wanted to bring him to my shop and put him to work, since he had a greater desire to learn than 75% of the people working for me.

Network Defender

Level 15

NT 4 ...OMG

Level 14

Yes.  Actually I have worked on NT 3.5, Novell 5.5, HPUX 7, Thicknet, and Unisys mainframes.  Old school nerd.

Level 14

I prefer .... "seasoned" "experienced".... (NT 3.5, DEC Pathworks, DECNET and VMS version 1 to ..... ) It sounds better.

Great posts network defender

Level 10

1. a co-worker deleted the user OU from AD on a Friday evening and didn't tell anyone until the next morning.
2. same co-worker unplugged the tape backup in the middle of a critical server restore and spent the next 4 hours waiting for the job to finish.

3. still the same co-worker instead of reverting to the previous snapshot after hosing a critical server, deleted the snapshot

4. yeah, still the same person, this time, opened a live server and installed memory... it was the primary DC

somehow that person stayed employed, finally quit one day in a huff for not getting a promotion.  last i knew they were out of work for over two years unable to get another admin job.

Level 14

Yikes..... A walking, talking, living, breathing disaster area.....

Level 14

I think I have met a few of this persons relatives.

Level 10

the excuse given for deleting the user OU was "my dog wanted to go for a walk and her tail hit the keyboard." the dog was blamed for a number of other things too... maybe this person was Les Nessman's cousin.

Level 10

the greatest NT 4 story now follows:

In 1999 I was living in NJ and working as a Computer Tech for a regional tech service company.  We had contracts with many Fortune 500's to maintain servers, workstations, and networking gear in satellite offices across central and northern NJ.  I got a call from one company IT person in NYC that the connect was down to the regional sales office and they weren't getting any email or access to the file server in the city.  I was already in the area so it was a 15 minute drive to the site.  I check out the leased lines, the demarc, everything looks OK in the switch closet... I checked the DC and saw it had a problem with the NIC... it had no connectivity.  I determined the NIC had failed and proceeded to shutdown the server to replace it.  It took 45 minutes to shutdown.  I replaced the NIC, started it back up, which took about 30 minutes to get to the login prompt.  Then I had to install the driver for the NIC which required another 45 minutes shutdown and 30 minute boot up wait.

I went and got lunch, ate it, and was back before it completed the shutdown... and it involved making a left hand turn, so you know it took awhile

Level 14

Me too!!! He has a large family....

Level 8

WOW!! I'm pretty sure your describing some of my EX coworkers! lol

Level 14

NT was an odd cat.  Saturday night was when we did our weekly maintenance.  Part of that maintenance was complete backups for all servers and then rebooting them.  We would also have to reboot the Cybex ATM backbone switches.  If you got the process out of order, the domain controllers would blue screen when rebooting the backbone switches.

MVP
MVP

I recently came across a set of 3.5" NT install diskettes...I thought, "I wonder ?", followed by not gonna do it and tossed them.

Level 14

Well played!!!

MVP
MVP

Now if had been OS/2 ?  Maybe.

Level 11

Dog can hit delete and confirm it with a swipe of the tail, eh?

Level 8

So we are working on a very important & timely project that requires many of our team members to be online researching and typing away at the keyboard while online (doesn't it seem this is when it usually happens). Anyhow, our internet access goes offline and we start to see what might have happened. Equipment failure? Cyber attack? ISP having issues? No one is in the datacenter by the time we go in to check the equipment which was fairly quick. By this time, the culprit had returned and confessed that he accidentally unplugged a cable on the switch that leads to the firewall device but plugged it back in. Unfortunately, the cable wasn't labeled so he chose a random available port on the switch which was turned off. Once the right port was discovered and the cable plugged in, we were back in business. It has made for a good laugh now that enough time has passed to heal the frustration in the moment. And we were reminded on the importance of a good label on the cable.

Scariest?  It's a tough one.  I've got three--you tell me which was worst:

1.  When I Needed A Get Out Of Jail Free Card    

getoutofjail.JPG
Maybe the time an IT Director at a different employer told me to move forward with a $65K test implementation of VoIP.  It was the first time it was being used in this chain of 33 buildings, 14,000 users.  No E911 solution, no T1's or PRI's, no dial-out if the WAN were down so no way to call for police, fire or ambulance.  He'd gotten a grant to do it, Cisco Call Manager version 1.0--buggy as all get out.

He wanted me to back him up with the C-Level and public-facing people, saying that was the right solution and there'd be no extra cost.

I ran the numbers and came up with $6.5M cost for a full implementation across all 33 buildings.  He told me that was not what he wanted the C-Level people to hear, and I was to say what he told me.

I got it in writing, showed it to my Manager, who backed me up but said "Hang on to that.  It's your 'Get Out Of Jail Free Card' when "IT" hits the fan."

Not wanting to be forced to lie to the C-Level and the public--even by omission--I found another employer.

Not long after, that  IT Director was "asked to resign" for numerous reasons.  This wasn't the only event where he was irrational.  It was scary working for him.

badboss.JPG

2.  When I Participated In A DDOS Against My Will

OutofControl.JPG

One day I saw odd patterns as I was watching Real Time Bandwidth gauges from the Engineer's Tool Set.  My corporate Internet bandwidth was ramping higher and higher--much more than ever before.

I started calling around, running tcpdump on the firewalls, checking the content proxy filter--what was causing the bandwidth utilization to get out of control?  Soon I had the ISP on the phone, asking them to filter out the NTP-1-2-3 attack I was experiencing--and they said they didn't do that kind of work!

Small external requests were causing my firewall to amplify traffic outbound at a specific South American business.  Killing my corporate Internet in the process.  It wasn't long before the Help Desk was on my back, my boss & HIS boss were in my cube, watching the bandwidth graphs hitting 800 Mb/s on a 200 Mb/s service.

More calls to escalate the filter request to the ISP eventually got to the right level, and although I'd correctly configured the firewall to discard the traffic, it was eating up all our bandwidth.  The ISP's upper level folks got their engineers to filter inbound UDP/TCP ports 123 and things got better immediately.  It seems multiple clients of theirs may have also been experiencing the same reflection attack participation.

It's tense when multiple levels of management are breathing down your neck and you've done all you know how.  Later I saw the Tech articles about that event--my company's Internet wasn't the only one involved--multiple NTP-vulnerable sites had been chosen for reflection attacks against that same South American company, and the Distributed DOS attack successfully overwhelmed their big/multiple Internet pipes and firewalls for more than a day.

3.  When An ISP Left The Barn Door Open And I Became Root

rootpower.JPG

Maybe scarier, in a different way, was when I opened up my Novell network browser back in the late '90's and found multiple new servers with print services available on my network.  On a subnet I didn't use internally.

On a hunch I tried pinging their .1 address and found a reply.  "So who's on my network?" I wondered.  I opened a telnet session to .1 to see if might be a router.  "Ah, a recognized router prompt!" says I.  "And it looks like a default router prompt for a 3Com Netbuilder Router--which I'm trained on.  I wonder . . ."

Well, the default user name was in place, along with the default password.  So now I'm root on someone else's router.  Let's see who the neighbors are . . .

Uh oh.  City government.  Police department.  Finance.  Who's the RIP neighbors?  State Government!  Not good.

Not wanting to probe any deeper, I got on the phone to that city's IT department and found there was no one there who was responsible for their routers.  They contract that out to a private company--the same one who was doing my company's WAN services!

OK.  I informed the City IT folks what I'd found and it went right over their heads.  I told them there was no security, default user names and passwords on their routers, and I could see their departments and their access into the State government networks.  They still had no clue this was bad.

So I called up my WAN service provider, asked them why that City network was spanning into mine.  Why the City had no security.

(mumble-mumble . . .  I'll get back to you shortly!)  and they hung up on me.

I watched and pretty soon those multiple City servers were no longer showing up in my Novell world, The oddball subnet disappeared.

Later I spoke informally with one of the Network Engineers for that provider, and he admitted "One of the guys spanned the City's VLAN into one of your trunked ports.  You both use RIP, you learned their routes.  You have security enabled and your ACL's prevented them from seeing you.  We dropped the ball for their security, both in VLAN port spanning and credentials not being changed.  Thanks for letting us know--and for not telling them!"

Maybe that's the scariest part.

confidential.JPG

Level 14

Item #1 hit close to home....but #3 scares me for mulitple personal reasons.

MVP
MVP

I feel old now. I started with Dos 3.2, then onto Novel 2.2 & 3.11 (became a CNE). Next was Windows for Workgroups before NT3.51 etc took over the world. Windows 95 was a real treat

Level 14

I think number one would be the scariest. 

Level 9

It was 1999. I had just been hired as a contractor for the Navy to manage a host of Oracle 9i databases, some containing highly sensitive data. Besides security at the base, the compound and the building entry, these systems were isolated behind four additional physical access controls within the building itself. Two of these were inside the data center proper—a perimeter cage and a soundproof enclosure. One of the systems had its own detail of two armed Marines guarding it 24/7. I needed four different badges of varying colors to navigate the maze of gates, doors and guard stations. There were even code words.

They were all open to the internet on the default Oracle listener port, 1521.

Wait, it gets better.

Day one, I asked the group manager for the privileged passwords for the SYSTEM and SYS accounts. She escorted me into her office, closed the door, picked up a Post-It notepad, wrote on it in felt pen (less impression on the paper), counted off several sheets and removed them (eliminating any trace of an impression on the pad). She dramatically placed the sheets in the palm of one hand and cupped the other over it. She announced that she could show me the passwords but I'd have to memorize them because the note will need to go in the burn bag. Immediately. It was as if the lingering scent of ink might give foreign agents a clue to access these systems.

I'd just visited with the Unix admin, also a contractor, who gave me the oracle Unix passwords. They were 20-something-character long and super-cryptic so I thought I prepared myself for something of equal difficulty.

She revealed the page and the passwords:

MANAGER and CHANGE_ON_INSTALL

For those not familiar with Oracle, those were the defaults for an Oracle 9i installation. CHANGE_ON_INSTALL was a reminder that the password should, around the time of, I don't know, maybe installation, get, uh, changed.

In their defense, they intended to change them but the user committee couldn't agree on anything so they chose to leave them at the default and revisit the problem later. Government at work.

Level 9

I told her that I needed to change them. Immediately. She said I couldn't. I asked why not. "Because the committee needs to approve it." I did it anyway and told her to fire me if she wanted.

Level 13

I just recently worked my first couple of off hours/ night work the other week. I work at a small college campus and everyone was in the dorms so walking to the building from my car was kinda eerie. I walked into the data center to start working on my upgrades. I was enjoying being alone until suddenly i heard something and there was some random guy behind me that I had never seen before, which scared the living day lights out of me. Found out he was part of the electrical dept on campus and he was working in one of the adjacent offices and saw the lights on and got suspicious too

Level 8

I was in applications at the time, but one of our tech support boys decided to make an untested on the fly change to the system.  Each transaction that changes the database writes a log record and those records are used by many offline reporting programs.  There are lots of different sizes of records, but not all record types have unique sizes.  And therein lies the problem.  The untested change wiped out a register and so the only valid piece of data in the log record was its size.  The entire rest of the record was hex zeros.  We spent weeks recreating those log records so our offline reporting would run correctly.

Nimda was a big deal for my company as well.  My manager and the CIO were out of state.  I was not in charge....I don't ever want that job anyway.  I went to the most senior staffer in IT after we got hit.  I told him that McAfee had a Stinger for it and it worked.

I asked him to round up all of the IT people that are in the building in 30 minutes.  I copied the Stinger to the network to a place that any user can get too and made it read only.

During this time I briefed the CEO of the issue and the plan to get rid of it.

Ten people showed up.  I showed everyone how to use the tool and told them where is was located.  I told them to put a sticker on the monitor to show that desktop was done.....and sent them out to kill Nimda.

Two went to the top floor to work there way down.  Two went to basement to work there way up.  Two went to the data center.  The last four drove in four different directions to take care of the branch offices.

In just four hours we had over 700 computers in twelve locations done and Nimda gone!

RT

Level 14

Well done... leadership comes in many forms...

I have had the number one issue before.....

Level 16

Guys, thank you so much for your stories! Points have been awarded for your efforts.

Level 14

Thank you maria.bungau‌.  This was fun.  Many good reads.

Level 16

network defender‌, so true. Whenever I need to brighten up my day, I just come to thwack and read comments.

It is like listening to Blues music.  You feel like this guy has real problems.....and my problems are getting smaller.

RT