cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

SysAdmin Horror Stories

Community Manager

*Cue wolf howling in the background

We’ve all heard spooky tales of hardware stored in rooms where the temperature was up to 140 degrees, servers drowning because someone left the window open during a rainstorm, and IT tickets that made the hair on the back of an admin's neck stand up..

Tell us about the strangest request you’ve ever gotten, the dumbest question a user ever asked, or those days when Murphy's law seemed to only apply to you.

Write a short SysAdmin horror story in the comment box below to complete this activity for the July Mission and earn 150 points!

285 Comments
Level 11

Dumbest question I've ever been asked:

"Can I use your [software installation] CD in my PlayStation?"

I also have a couple of dumb user responses to what I thought were sensible questions:

"What version of Windows do you have?" "Double glazed"

"Can you please close all the windows you have open so we can start again?" *sound of user walking away from his desk, and physically shutting the window*

"Can you click on the icon on your desktop?" "No, because my computer is on my table, not a desk"

One of my more memorable horror shows...

Once upon a time there was a user named...let's call them Bobby.

Bobby processed a lot of documents and approvals through a proprietary system and as you would expect; this system would time-stamp and tag any updates with Bobby's userID. Simple enough.

One day I get a call from Bobby, raging, that he can't print and can't take screen captures anymore. Performing the usual printer queue fixes, there's a number of documents in there, like 15. It gets cleared out and seems to be printing fine. I check his screen capturing tool that seems ancient to me but whatever; it works for him and it seems to be functioning normally now. He's happy. Unable to replicate the issue, Ticket closed.

2 days later - same thing happens, I get the call a little later in the day. Same actions to fix it except there are more documents caught up this time, like 20-25. After clearing the queue, I update the printer drivers and make sure this workstation isn't missing any needed patches. Can't seem to replicate it again with new drivers, or printing subsequent screen shots. I offer to replace the 10-ish year old screen capture software with a new open source software or we can order a COTS that some other people use - " Its out of the question to change screen capturing software!" he barked. Unable to replicate the issue, Ticket closed?

A day later - it happens again. Now I'm frustrated and in it till something gives up the ghost here. I schedule to sit with Bobby for his whole morning and lo and behold Bobby is doing some curious actions. Bobby is screen shooting every step of the approval system as he works...and is printing them. Creating multi-page screen-shot documents and then printing them, and moving on to the next document or approval which can sometimes be 8-10 pages long and since a single page is larger than the screen capturing tool that means multi-caps per page so thats more like 17-20 captures and image prints. Sure enough we get 10-12 deep and things go haywire. The screen capping software stops functioning and everything stops printing.

  • Me:
    • "Bobby, can you tell me why you're screen capping everything and printing it? Is that part of the business process to approve these things? (I have no clue)"

  • Bobby:
    • "Oh no. I do this to cover myself and prove I was working so when 'they' say I didn't do something I have the proof!"

Bobby proceeds to open his bottom desk drawer and it is full to the brim of these types of screen shots going back more than a year. I mean we're talking like a 5'x2' desk drawer here people.

Calmly explaining to Bobby my doubt anyone is out to get him and that even if they were - the system he's using locks the data by user-role, time-stamps the data state changes, and tags it with his userID and is a much more encompassing way to track his work than screen caps and physical copies that cost money via paper and toner while breaking his print queue or screen capturing software.

  • Bobby:
    • "That unacceptable! This HAS to work! Its ALWAYS worked!"

At this point, Bobby was less than happy with me, and to reference him as irate - would be putting it mildly and generously.

  • Me:
    • "Ok, can you explain to me or show me how this relates to the business process? Does everyone screenshot all of their work, everyday?"

This couldn't be true but I had to know and as you might've expected - Bobby could not cite documentation for this nor anyone else that did this. No one in the department. Not a single one - of 14.

I felt bad for Bobby so I told him maybe I could work something out.

Grabbed a meeting with Bobby's boss - whom was starting to wonder why I was spending so much time in her department, explained the technical problem and then the actions that caused it. She hit the roof when I explained about the ginormous desk o' paper lurking in that cubicle.

Turns out, Bobby's boss; lets call her Jennifer, was becoming increasingly concerned with Bobby's performance - doing things that should've have taken so long were - and I suggested cutting out the screen capturing and of course it was shortly thereafter.

You might be reading this going, that's not a horror story - look it even has a happy ending that saved the company money and probably helped an employee's performance! What's horrific about that?

3 little things:

  1. I had to deal with a printer and printer drivers. Eww.
  2. I had to deal with software that was 10 years old that I have never used and seemingly buggy.
  3. I had to deal with an angry user. Oi vey!

Those things are horrific to me.

Level 12

There once was a happy web master with a web development change process that included help desk software, subversion "SVN" with commented changes that matched the ticket number. This helped to document who authorized the changes to the website, each department manager was in charge of approval of their section. All was peaceful for 2 whole years. Until thousands of change requests later. One manager became upset that a change had been made to their section without seemingly any authorization, she was going for skin escalating all the way to the CIO.

The web master pulled out his trusty blame tool and begin to analyze the code change in the SVN, which showed the ticket number in the change comments that authorized the change he had made over a year ago. Reviewing this tickets information in the web help desk software he was able identify that this very manager had authorized the change, took a screenshot of the the ticket information and e-mail the proof to his manager, CIO, and the accusing user "Awkward."

This story had a happy SysAdmin ending, so I suggest you plan ahead, create a change process that will protect your skin, and possibly the livelihood that you enjoy so much. There are plenty of tools available the best ones how ever are those that track who changed what, and add comments to back up those changes such as the ticket number.

I was just starting my IT career, and I was told a VIP user couldn't VPN in, and I was asked to help. Everything checked out with the computer, so I asked the user to try it in front of me. He took out his RSA token, knew what to do with it, and it worked.

I also knew this user had been complaining of this issue for some time, and I wasn't the first person to try to fix this. Something wasn't right.

I asked him to walk me through every step he took from when it failed the night before.

"Sure, I get out my laptop, plug in the network cable, get on the internet from home. I start the VPN client, take out this paper with the code on it, and type it in..." Yup. He wrote down the RSA token's code before he went home. See that little thing was expensive, and he didn't want to lose it. I explained that the number changes all time, and that he needed to have it with him. VPN issue resolved.

MVP
MVP

Me: Can you plug in the network cable into the computer then into the wall jack.

User: It is plugged in.

Me: Make sure it is plugged into the one that has a D for Data Next to it.

User: It is.

Me: Ok I am not seeing any connection on the switch. There must not be a cable from the patch panel to the switch. Can you go back to the Network closet and plug in a cable from the Data Patch Panel to port 5 in the switch?

User: Ok doing that now.

User: Ok, the cable is plugged in.

Me: Still seeing no connection. Can you send me pictures of the wall jack, and the patch panels.

User: Ok.

After receiving the pictures it was noticed the user instead of plugging in the cable on the Data Patch panel (Clearly labeled), they plug the cable into the Voice Patch panel (Clearly labeled).

Level 14

While travelling to a new client site I received a call asking how long I would be because the machine room was on fire. I told them to call the fire brigade immediately & that I would be on-site within the hour. They chose not to call the brigade, or do anything else, because they assumed that I would be able to fix it when I arrived.

In fact it turned out to be a failed air-con unit that was venting. The remaining a/c plant was seriously under-powered. The room was very hot, the corridors outside were hot & smelly, but being a rounded techie, I handled it. The a/c units were replaced with appropriate kit, not repaired, the aging failed servers were fixed or replaced and monitoring and alerting was installed – A good result all round.

Level 12

hi,  this is about a seminar  i was going to start it at moment user said me to change some settings in system. i have done changed settings according user.

after that system is not ready to start. i was shocked. user is very chilly. now how i can start it. this is a very dangerous problem created by me for me. that seminar was not going good. that situation was very horror for me.

Level 9

I used to work for a very small company in a typical office park environment.  We were having connectivity issues, and I was trying to trace the cables, and they all led into the floor.  I asked what was under the building so I can check on the wiring, and was told there was a crawl space.  I asked how to get down there, and was shown a hatch in one of the rooms.  When we opened it, it was like staring into the pit of hell.  It was a dirt floor 3-4 feet down, and covered in 12 inches of water.  I could see wiring laying in the water, and garbage floating in it.  I had to bring a pump in, and pump water out for days before I could even think of going down there.  It was horrible.  Totally cramped, smelled terrible, muddy, etc.  I was amazed anything worked there.

Level 11

Going back to my days as internet help desk I would have to ask customers which browser they used (it was usually IE or Netscape), and the typical answer was "What's that?".

One of my worst days was self inflicted.  I had spent hours over night making changes to a router that supplied dial-up lines for internet services.  The next day (for reasons I no longer remember) the router had to be restarted.  Come to find out I didn't write the config nor did I save a copy and I spent the better part of a day working to get it all corrected all while handling angry phone calls and evil stares.

MVP
MVP

Had a user complain that the computer was eating the CDs inserted. The user claimed they put a CD in the slot but when Eject was clicked there was no CD in the tray. Turns out the user was sliding the CDs in between the CD ROM casing and the chassis. When I opened the case I found several disks just laying in the case. Yikes.

One of the departments we supported at an old job was the university police.

My colleague gets a call from the police chief, letting us know that there was 'something wrong' with the ethernet connection on his recently-issued laptop and he can't get on the network.

We take a quick ride over to have a look at the laptop in question, and the connection issue root cause was evident.

Somehow - and I'm still not sure after all these years how he did it - but the chief had forced the ethernet cable into the RJ-11 modem jack on his laptop. Inserted it so powerfully, in fact, it had cracked the laptop case around the connection point.

That'll teach it!

Level 8

Probably the worst horror story I can remember aside from normal user headaches was having to do research on a user's web activity. This user had shown to be causing some spikes in the afternoon that actually slowed the location that they were at. After a few moments of study I noticed the name of one of his heavily visited sites. I don't remember the actual name but it was a site dedicated to large women. I took this information to my supervisor who passed it up the line. I received word back from my boss's boss that they needed examples of what types of sites he was going to, with screenshots. The rest of my day I will never be able to forget. Had there been bleach near by I would have flooded my ocular cavities early in the search. Apparently the "gentleman" in question spent 80% of his time browsing very questionable material in a shared office. So after 2 or 3 hours of gathering screenshots I contacted my boss's boss ready to deliver my research. He casually told me they'd confronted the employee and he had quit on the spot. My research was no longer necessary.

Level 9

Doing a weekend UPS timing (i.e. how long the battery will last until 25% full) in our DC and watching as the percentage available closed in on 25% (note, we decided to make the test "better" we'd switch off power to the room . So as we admins watched the power drift to 26% and prepared to power the room back, it suddenly went to zero and powered off 10 42U racks (containing production network, SAN and physical server gear). After looking at each other for 10 seconds in disbelief we slowly brought everything back up (amazingly with only  hard drive lost).

Needless to say, going forward we did not perform a similar test....

Level 14

Back to dumb terminal days. (Circa 1986)  I'm a VMS Sysadmin.

We were moving a group of application programmers from one side of the floor to the other. Not an uncommon experience in this place I worked. Friday comes and I walk around an remind not to power off their VT220's as they were getting new ones. I get to one cubicle and I say to one of our programmers..........."Pat"

Me: All set "Pat"? See you on Monday.

Pat: Great..you too.... but can I ask you a question?

Me: Sure, What's up?

Pat: We're getting new VT's right?

Me: Yes

Pat: How is the information on this VT going to get to my new VT over there?

Me: (thinking quickly and trying not bust out laughing).... Don't worry, I'll make sure everything is there!

Pat: Thank's I really appreciate it, there's so much stuff, I don't want to loose any.

Me: Have a nice weekend.....

Saw my boss about 20 minutes later and relayed the story....we both enjoyed a good laugh.

Moral of the story..... think fast, have a sense of humor and laugh later.

MVP
MVP

I once did a SolarWinds upgrade (two pollers, NPM/ SAM/ VNQM/ IPAM/ NCM) that I intended to start around 4 PM.  External issues led to management rescheduling me for 6 PM.  Every last piece of installation software seemed hosed in some way.  Upon double-clicking, each one was immediately finished without seeming to have done anything.  I did a great deal of downloading software again (thinking install software was corrupt) and worked with SolarWinds support.  We scoured logs and captured diagnostics.  This continued overnight and before I knew it my coworkers were coming back in.  SolarWinds support stayed with me along the way and I worked with folks on at least three different continents.  Eventually a coworker pointed out that some of our security software was working as designed and was stopping my installation.  I believe it was close to noon into the next day and I was sent home due to having been onsite for about 30 hours at that point.  My coworkers stepped in and completed the upgrade without any issues after HIPS was disabled (despite being SolarWinds n00bs).  Needless to say I am vigilant now to involve personnel controlling host intrusion prevention software.  Have mysteries in your SolarWinds installation?  Running host intrusion prevention software?  Lots of opportunities for pain.

Level 12

In days of yore, I was asked to drive from one side of the city to another to assist a client with a printer problem.  I drove out to the site, opened up the printer, and removed a pair of scissors that was wedged inside between the bar and the tractor feed.  Plugged the printer in, worked like a champ, have a nice day.

It only took me once to mess it up, but I remember to always double check now. I was rolling out a new VLAN on our network that would be used in a certain wing of the building. I had it all setup on the distribution switches and just needed to added it across the trunk link since we were only allowing the VLAN's in that area that were being used. Well I bet you can see where this is going, switchport trunk allow vlan XXX, not switchport trunk allow vlan ADD XXX. DOH!!! So after locking myself out of that switch I ran to the closet that it was in with a laptop and corrected my mistake. Thankfully they were only down for about 10 minutes, but that was a 10 minute outage that didn't need to happen.

Level 16

My favorite is fairly simple.  "The internet is down....  I sent an email to my cousin in Florida and they replied that they know the internet is down.  When will you guys fix the internet?"

also...

A call to the HD late at night....

User: My computer doesn't work at all

HD: So you are saying that no applications work on your computer?

User: Yes, I was kicked off the network.

HD: I see that you were browsing the internet about an hour ago, is this correct?

User: Yes I was working and then it all stopped.

HD: Did you receive any 'red screens' while you were browsing the internet?

User: no...

HD: Well, we see here that you have been visiting pornographic sites over the last few hours with over 1000 visits to 10 different internet sites.  This caused our security appliance to automatically block your connections to the outside world and send alerts to management.  So, you said that you were working just fine about an hour ago?

User: What?  I must have a virus!

Level 12

User called me on the phone. Had an issue with PC. Asked them to explain more and they said "look I am pointing at the error on the screen".

Level 12

I had a student once who asked me, "Why won't my computer boot up anymore?".  Seems innocuous at the onset, but after a few questions I found out that he had limited disk space so he started deleting files based upon their creation and used last date.  Do you see where this is going?  It was Win95 and he deleted the ini and inf files so that upon reboot, the OS would not load.  OMG, funny!  Had to drop back and restart that learning process and educate him on all of the OS files that cannot be touched else once rebooted an OS will not load.

A few more for your amusement...

Worst Computer service calls

Level 13

Back when I was interning as one of a total of 3 (including myself) at my local city government, and the water department decided they wanted to put a wireless router in their building. So my boss's assistant and I go over on a Friday to hook up the router without any real modifications to the settings and just left it because everything seemed to be working. During the weekend he got married and the next week he was off to go on his honeymoon. Once I came in on Monday my boss talks to me about a problem at the water dept saying they have some sort of networking problem.(note I was/am still pretty new at networking). I headed over and found out that for whatever reason, people would be getting kicked off the network, even the computers that were hooked up to the switch via ethernet cable.

After doing a bit of troubleshooting and talking to my boss about the problem, the next day my boss thought maybe the switch was starting to go bad as it was pretty old anyways. So the whole next day I had to re-mark the wires and tried to properly setup the switch (I finally understood why marking where the cables were going to was so important >.>). Wensday and Thurs when by and I was still scratching my head about what was going on. On Friday I decided to check the router and some of the setting (I think this is what happened) had been messing with the network by kicking off people on the network and reassigning the ips to other devices that were being added to the wifi. Come Monday there didn't seem like there was a problem but, my boss's assistant came back and checked it out and reconfig the router and everything has been going well. Probably the most stressful/strangely entertaining week of my time interning there

Level 9

  Years ago our department had a VAX 11/780 computer room. It had a Britton Lee database, large physical old style top load canister hard drive platter stacks and a reel-to-reel tape drive back up cabinet. I worked second shift and volunteered to act as part time computer operator for them. This included running the nightly backup routines which required a couple of tape reel changes. Never a problem. After a year or two a young lady promoted up through the ranks wanted to learn some computer operations and volunteered to help me. Well, about two to three times a week we started having issues with one of the disk drives gong off line. This in turn caused the backup up operation to fail and I had to get the disk back on line and mounted and restart the backup jobs.

  After a while I realized that this never happened when I was doing the backup job, but only when she did it. So I discretely decided that I would feign being busy doing something in the computer room when she came in to do backups. The cause of the recurring problem was soon obvious.

  The disk cabinet that always went offline was directly across and facing the tape drive cabinet. After the first tape was full and needed to be swapped she would pull the full tape off. Get a blank one, put it on the spindle and then bend at the waist to thread the tape. As she did this her rear end would hit one of the buttons on the disk cabinet and take it offline !

  That solved that problem for good.

Level 9

I performed an e-mail migration from a hosted exchange managed by a 3rd party to Office 365 and 1 of the users couldn't log in the next morning.  I asked her if she reset her password using the instructions provided and/or tried the temporary password sent to her.  She said no and complained that 6 months ago (prior to my hire) she got a virus on her computer twice and it was unfair that this was happening to her a third time.  I reminded her that this was an e-mail migration, not a virus on her computer not allowing her to login.

I watched as she tried 3 times to enter the password provided, she got it the 3rd time.  She was now prompted to reset her password and failed multiple times.  Apparently she was trying to use her sons name "alexander" as her password.  I informed her that it was not a strong password and cannot be used.  She then began screaming at me (she was a Vice President) that the e-mail migration needs to be undone as she doesn't want to learn a new password as her banking password also has this restriction and she does not want to go through that again.

I informed her that I can open a ticket with Microsoft to change their password policy, but until they do that, she will have no access to e-mail.  We negotiated that she would set her password to "Alexander1" and she them took a marker and wrote the password in big letters on her wall to prove to me that the password is not secure.

The user was fired for unrelated reasons 1-2 months later.

Level 9

EU: My computer is not working?

HD: what do you mean?

EU: Screen is black but has an amber looking light on the monitor, it has never done that before.

HD: Is the computer powered on?

EU: yes.

HD: Are you sure?

EU: yes, I'm positive!

HD: Okay, hold on.

**HD walks over to investigate. Machine was powered off.**    /facepalm

Level 9

It's not one stupid moment, but rather a collection of similar ones.  An end user will complain that a function in software isn't working, or a second monitor is no longer working.  I pay a visit to the desk to check cables or logs, only to find out that the user hasn't even attempted a reboot.  All of a sudden, everything is working as it should.  Oh, the miracles of letting the computer reload everything first!

Level 8

If you want a real long, true horror story about a wireless deployment:

http://community.spiceworks.com/topic/531913-the-redeployening-a-true-networking-horror-story-why-oh...

But the daily horrors are the most grinding:

A department manager who, three times over the course of a year, called IS in a panic because his computer is "dead", when in reality, his monitor power button was off.

An end-user who asks for help because they "couldn't understand the instructions", only to find the original email with instructions unread in the deleted folder (needed a link off it to complete the install). Thanks for not trying! Or even reading my email to you!

A grown man who not only doesn't know how to format an email address, but literally can't find the @ symbol or even a the period key on the keyboard, and needed the exact location of these items pointed out EACH TIME.

The horror, the horror...

Level 7

Had to travel to a remote office to upgrade a few users from old Dell Dimensions to some slick new Latitude laptops.  Data migrations went very smoothly, more than expected, and the users were happy so I went back to the home office. A few days later, I get an email from one of the users that they used to be able to put CD's into their desktops and play music, however they told me that they did not fit in the laptop CD trays.  Apparently, they were not putting the CD securely on the spindle, and were then forcing the tray closed. Fortunately once they found that they had to use just a bit of effort to mount it properly, they were once again happy listening to their tunes.

Level 12

Dumbest question:

I was working in a Data Center where we provided managed co-location services, not hosting or hardware.  Client grabs me and takes me over to his cab saying they have problem with something in their rack.  Points at an ASA in the rack and asks

"Can you help me guess the password for this firewall?  I forgot it."

Level 12

Several years ago I worked for a fairly large software company in Bellevue, WA (no, not the one that is located in Redmond, WA) ...

This particular day was like any other, I arrived early, did my normal routine, put out some fires and then went home.  After dinner, my wife and I decided to go for a drive and take a walk around one of the newer neighborhoods in our area.  I was on-call that evening and received a  page (yes, this was when pagers were still the norm).  When I called in to find out what was going on, I was informed that we had lost power to our entire data center.  Needless to say, the nice, quiet walk with my wife would have to be postponed.  When I arrived at the data center and started turning on servers (some of which had not been powered off for years!) some of the servers came up just fine and we were back in business while others would not power back on, some made it to the POST with hard disk errors.  It took us all night and into the morning to bring our data center back online.  The cause for the power outage? ...

The building that I worked in had a security guard on duty after business hours that would do hourly rounds and visit each floor to insure that everything was fine (no intruders, etc.).  The data center was located in the basement and when everyone in our department had left for the day, we would always set the alarm.  Well .... this particular day there was a "substitute" security guard.  First time at our building.  When he arrived at the basement and unlocked the doors to our area he triggered the alarm.  The alarm beeped a few times indicating that he would need to put in a code to disarm the alarm or trouble would insue ... he did not know the code so after about 10 seconds the alarm started blaring.  The security guard panicked and looked for a way to silence the alarm.  He noticed a big red button on the wall just a few feet from where he was.  He immediately ran over to the button and pressed it.  There was immediate silence.  The alarm stopped blaring, the computers stopped humming, the monitors stopped glowing.  He had just used the emergency power shutoff button to silence the alarm. 


At the post mortem meeting, we installed a glass case around the red button and clearly labeled it 'EMERGENCY POWER SHUTOFF'.  Those of us that came in that evening were also given T-shirts with big red buttons on the front and some wise saying on the back (I can't remember what the words on the back said, but it was pretty clever).  Now any time I see one of those big, red buttons I think back to that day in the late 1990's and it makes me smile.

Level 14

had similar thing happen with an HVAC tech that confused the BLACK button that got pushed to exit the room with the RED button clearly marked EMERGENCY POWER OFF. Clear plastic cover installed with in 24 hours.... after 3 hours of recovery!

PS... He told his boss that he did not do it.... the camera that focused on the door told a much different story. He was persona non grata at our site after that.

Level 14

So many horror stories, but I think the winner has to be:

A user called in when I was a green helpdesk tech at a call center and informed me that when she turned on her monitor it began to spark and smoke.  I wasn't so green that I couldn't tell her to try to safely turn it off and unplug it if possible and made sure that it wasn't on fire before sending the case to hardware support to replace the monitor.

That's my own personal, I collect these so I have even better from friends.  I've just never been so directly involved that I was the guy dealing with the server room fire while they had locked themselves out.

Level 15

A company I worked with had a fancy conference room that looked through a window into a server room full.  About 4 full-sized racks with Cisco, Avaya and Mitel equipment that they used for demos for customers, because blinky lights are cool.

This server room had it's own A/C unit, and an ancient system for sending texts to a group of techs in the event that the hardware got too warm.  I was not in this group, and had no idea how to access, configure or control the alerting system.  It sure wasn't SAM/NPM!

On this day, the A/C had packed up, and the temperature was north of triple digits.  As a lowly front-line tech, I was in the basement NOC answering phones, unaware that the fancy demo units were baking.  The three people who were supposed to get alerts were outside cell coverage, on vacation, and out of state for a trade show, so none of them were aware of or responding to the alerts.  I am taken by surprise when the CEO comes to my desk and chews me out for an hour about my negligence of his equipment (that wasn't my responsibility), my failure to respond to alerts (that I didn't get) and my failure to add myself to the alerting system (that I didn't know existed until that moment or how to configure or access).  Also, I'm not an A/C repair tech.  I attempted to explain this while working to shutdown what I could, but in order to maximize blinky lights, we had production equipment in the racks as well and you can't just take the whole network down in the middle of the day, right?  Then we called an HVAC guy and propped doors open while we waited for him to arrive.

The CEO was having none of it, and I was "laid-off" shortly thereafter.  They didn't say it was because of my failure to respond to this "emergency," but I'm pretty sure this was a factor.

Level 9

Once I was asked to assist a user who was having issues transferring files up to a website on the internet.  After some time of troubleshooting over our network I asked if she would be available to come by our office to try the same task over an outside network DSL connection that we use for isolating possible routing issues.  As she arrived she set down her laptop, opened it up and hit the power button to start it up.  I thought it interesting that it was shut down considering she works within the same building and we had just spoken over the phone while she was sitting at her desk.  Anyway, she proceeded to turn on the laptop and noted that the wireless was a lot better in our basement office as her laptop booted up much quicker than normal.   This was one of those times when I knew that any efforts of mine to explain anything IT related would be futile.  I smiled and nodded with a "huh".  Now that the laptop had booted up, we were able to test the file upload across our DSL connection to find the same result confirming that we were seeing issues with the website we were trying to upload to.  Once we had completed our troubleshooting we said our farewells and she proceeded to hold the power button down for a hard shutdown of her laptop in preparation for her trek back upstairs to her office where the laptop will undoubtedly go through a much slower boot up due to the inferior wireless supporting her office area.

Level 11

Well I had a coworker in school, we were apart of the IT helpdesk and he told me that, "Hey if you just copy Microsoft off of this computer and put it on a flash, you can copy it to all the other computers and it works quicker than the CD".

When I realized what he was trying to tell me I said, "Hey bud, have you noticed that it has a little curly arrow on the icon? " " Have you ever wondered why its size is like 1 KB?"

Coworker: Well yeah, but that doesn't matter, I know it will work.

My head hits the desk. I didn't tell him either, always so sure he was right all the time. I figured I would let him learn a few life lessons. He was one of the guys that helped students and staff when they came in with technical problems. Those poor people, I caught as many as I could before they left. When I told my supervisor he just said, "I can't do anything about it, he is a warm body."

I asked about some training, but he said he didn't have the time to teach them.

Level 9

I was working at one of our smaller offices, on a cabling project.

One of our CADD users was working with his office door closed. I was pulling cable and had some ceiling tiles pulled - could hear him cussing and crumpling paper over and over all afternoon.  At the end of the day, there was an 11x17 sign on his door printed with one of the CADD fonts  - "Whoever stole my Johnny Cash CD, please return it." 

Went into his office to do some termination - his garbage can was stuffed with botched attempts at the sign.....he had spent the whole afternoon on this project.

Level 11

I had a new hire/trainee for the hospital Data Center operations tech job, only a week into her new job. We were supposed to be at another campus for a staff meeting and it was canceled at the last minute, which turned out to be a good thing. As we were standing in our office just outside the Data Center, she stops me and says, it that light fixture supposed to be dripping water? I looked, then peered into the Data Center door window and saw it literally raining in the Data Center. One server rack was saved by a keyboard that was incidentally left on top of a system and it was collecting the water that was dripping into the rack. After calling in the cavalry and a bucket brigade, we found out that the labor and delivery unit upstairs was being remodeled and they pulled off the job-experienced plumbers for another job and sent in a replacement who had never worked the site before. He uncapped what he thought was a dead water line to add a new fixture and it turns out that line was on a separate main and shut-off valve. The floor above us had just been completely remodeled and now had about four inches of standing water and we were in a slight indoor rainstorm below it. This isn't even the first time we had water in the Data Center in the ten+ years it has been where it is now. Fortunately due to other more pressing matters, we're slated to get a new DC in the next year with nothing located above it.

Level 7

Thankfully most of my dealings are with other techs/engineer's from service providers.  Every so often I have to deal with a customer.  Recently I was working on an issue for a customer and requested right out of the gate for him to power cycle our CPE.  He claims to have done so but my monitoring system never indicated a power cycle but the assure me that this was done.

My company provides telecom services so I open a trouble ticket with the local provider.  At that point Murphy proceeds to cause me much pain.  The provider claims to see an issue on their network and dispatches out to the field.  The provider claims the issue is fixed but I show it down.  For the next 2 full days the ticket goes back and forth testing closer and closer to my colo facility.  Still no resolution and the customer is rightfully getting upset with the delay.

I had to dispatch one of our technicians to our colo facility to help me.  Surprisingly everything is testing fine.  The tech tells me he is 10 minutes from the customers site and will call me back.  I notice the circuit come back up in service.  I get a call from my tech about 5 minutes later to let me know they powered off the CPE and never turned it back on.

Sadly multiple tech's over multiple days wasting multiple hours.  Even sadder the person that was supposed to power cycle the CPE was not a secretary but the Sys Admin for the location.  On days like these I am comforted with the fact that I will always have a job.

Level 10

Oh boy...

As a consultant, I have seen some horrific things. One of my clients had retail stores across the U.S. that specialized in jewelry making. I traveled to many of their stores to clean up their server rooms and replace antiquated hardware. Many of their servers will still running Server 2003 (this was in 2011) and running both desktop and server hardware that was 10-13 years old. I went to one particular store in the middle of summer in a notoriously hot and humid state to replace their servers. They had a primary and secondary; but both were fond of having their fans fail, hard drives die, etc, and were always running at high CPU usage on SolarWinds reports. When I arrived on site and walked into the room, I wanted to scream. Their jewelers room (where they shave down metals and clean pieces) ventilation was incorrectly routed/broken. All of the soot and shavings from the jeweler's shop was blowing directly.into.the.SERVER.ROOM. This had been going on for years; and due to their lack of IT budget for years/ IT being at HQ many states away/poor management, this was not reported. I estimated this cost their location thousands a year in repairs, replacement equipment cost, and downtime. Everything in that room was covered in an inch of black soot/shavings, and it was not easy to breathe, so I walked out.

Needless to say I immediately called HQ and told them what was going on. They had an HVAC tech on site next day to fix the ventilation and clean up the mess, and I refused go back into the room until this was done. No way in hell was I going to risk my health to put in brand new equipment -- not to mention it would just get ruined anyway. In the rest of my time working with that client, that location never again had a server hardware failure. Just goes to show what a few accumulated years of "We will deal with it later" can add up to.

Level 15

The one that I always refer back to is a former coworker that spent a good 5 minutes arguing with an instructor that ICMP used UDP Port 161... He graduated to customer within a month or so after that one. There were a multitude of transgressions, but that argument is sealed in my mind forever as one of those "Whuuuuuuuuttttt????" moments.

Level 9

My story is about required processes...Need to add DHCP entries to the DHCP server. Here is the process. Receive request. Write 5 page document (no exaggeration) detailing who submitted the request, why the request was submitted, what the solution would be, the detailed steps of the solution including spreadsheet showing how each field would be completed and backup procedures. Produce second document to include pre execution test plan, and post execution test plan in minute detail. Submit to CAB board for review, submit to higher level advisory board for review; attend CAB meeting for formal approval; attend additional approval board meeting if data center is in freeze; attend post implementation board for lessons learned...Lesson learned: now I know where our tax dollars go...

MVP
MVP

While working at a bank in data comm department, we used to be on call for a week at a time.  My coworker was actually on call one night but was probably drunk and incoherent again, so I got a call.  My faminly and I happened to be staying at my mother-in-law's house out of town that night.  It was a small house and I thought the polite thing to do would be to go to the car with the cell phone and laptop and VPN from there rather than wake up the rest of the house with my commotion.  A few hours later I let myself back in the house and was heading back toward the bedroom when I heard the unmistakeable sound of a shotgun click.  I believe she said "who are you"...or something to that effect.  She didn't shoot me (resisted somehow) so I'm here to relay this.  The worst part was that it was not my week on call...it was my lush coworker's week.

Level 10

The A/C story rings true to me and my team. We had a data center get so got (how hot was it?); it was so hot that our newbie engineer, when sent in to power off everything (and he took this too literally while we were driving to the site) - the soles of his shoes MELTED.

Instead of hitting the EPO switch, he went to each cabinet and powered off the PDU - a little more work than required. Of course, when we got there we had to give him a hard time. He was sweating, had his shirt unbuttoned all the way. One of the other engineers asked him "Where's your tie"? We all laughed at that one (our employer had a strict dress code).

Level 9

This happened many years ago. The building where I worked had lost power during a storm in the evening. The user PC's were not on battery backups, so all the user PC's where off in the morning when they came in. One user called me and claimed she couldn't login. I told her the power was out the night before and she would need to power on the PC. She claimed she did that. I walked over and saw the "No Signal" message on the monitor. I asked again how she turned on the computer. She reached across her desk and turned the monitor off and on again, then looked at me like I was the "dummy". So I reached under the desk and pressed the power button on the tower and magically everything worked. She was dumbfounded and had no idea the computer was made up of 2 separate pieces.

Level 9

One of my biggest terrors was working with the some folks from the engineering team at one of my jobs. They would constantly make changes without going through an approval/change process, and most of the time they would download huge install files and proceed to save gig's of log data.  This would be all fine and dandy, but they would do this on systems that were in production, and all those logs they generated, they would never clean up.  This resulted in a request to increase storage space on various servers about every 3 - 4 weeks (and these weren't critical/compliance related logs, so there was no need to keep them).  Eventually we just got tired of it and told them they had to clean up their systems, or we would do it for them.  One script later, we cleaned up their systems for them, because they never took the time to do it.

MVP
MVP

At a previous job the power went out.  Not a huge deal, the batteries and generator kicked in just fine, so everything is good, right?   Well, it happens to be mid-summer and when the power was off for the extended time, we couldn't run both the cooling systems and the servers on the generator.   The temperature quickly rose in that room to very high temps and we had to open all the doors to the facility and bring in huge fans to try and keep the temperature down.   Poor planning on the power-side of things!!

I also remember during the east-coast blackout we had problems with our WAN link back to our company HQ in Germany.  Even though we had purchased dual-T3's from different vendors and went to the trouble of making sure they went across the Atlantic at different points, one was supposed to cross the Atlantic near New York ,the other way down south, at some point the second vendor had re-engineered the circuit and it got redirected across the same fiber bundle that the first circuit was on.   The place where that bundled was fed on our side went down part way into the power outage and we were cut-off from EMEA completely...

Level 11


User calls an complains their printer is jamming and they don't know why. I send tech out and he finds the toner silica packet wedged in the fuser. Now gimme 150.

MVP
MVP

User calls to say their printer is not printing and is making a beeping noise...

Arrive, look at it, ask where they keep their paper, reload printer and away it goes...

The look of horror that they overlooked that possibility was priceless.

Level 12

Many years ago while working at a Printing company, our cable plant was Thin-net (co-ax cable with T-connectors for each pc). Things were going great for awhile, then one day at about noon, the entire west end of the very large building went down. Started trouble-shooting and suddenly about a half-hour later the network came back up. Continued to look but never found anything. Next day, about noon, same thing happens, west end goes down, comes back up about 30 minutes later. Cannot find anything wrong. This goes on for about two weeks, driving me crazy! So I start checking and rechecking cables and connectors. Get to the office of the Pressroom supervisor about 11:45 and tell his administrative assistant I need to check her computer. She says "ok, I'll just take my lunch break now", and proceeds to lean back in her chair and hook her feet on the co-ax cable hanging behind her desk like a foot-stool. The west end of the building immediately goes down. I ask "how long have you been doing that?" She says "every day since I started, about two weeks ago." She was putting just enough pressure on cable to break the circuit, but not enough to pull it apart, and when she put her feet down the cable would go back into place. And you thought users were only dangerous when they had a keyboard!

Level 9

I have had plenty that I can't recall right now but there is this one case I got where the end user reported that a duck was on her computer monitor.  1st level support checked her machine out (remotely) and didn't find malware or an app that could be causing this behavior.  They send me the case to have me check it out (physically) but before she could bring her machine in she stated the issue was resolved.  I was very curious as to what caused this anomaly so I inquired; she said that a duck sticker had been blown on to her monitor by the fan and she just noticed it.   

thumb_COLOURBOX7085650.jpg

Level 9

I have many crazy things happened working in IT but this one is one of the worst and it is not even an end-user.   This was a new hire and he was hired with the same level as me.  He said he has over 12 years of networking.  He asked me help him look into a server that was reporting down in our monitoring tool. I asked him did he ping it to see if anyone changed the IP address.  He said yes. I went to his desk and asked him to ping it again.  He opened “Putty” and typed in “ping servername” in the Host Name field.