Wednesday, October 24, 2012

IT nightmares

These are things I have experienced that make me shake my head in disbelief.
  • Why did the power go out in the server room?
    • Because someone plugged a fan in.
  • I can't ssh to my current production server, what is going on?
    • Uh, It was accidentally decommissioned
  • Is this decommission ticket correct?
    • Ah yes, I put it in 8 months ago and it indicates it was handled and closed
  •  What happened - why did the network act up?
    • Ah, a network cable came unplugged.
  • What do you mean the manager is having a large scale demo today?  The application hasn't been tested with more than two users.
  • He wants the change in production without being tested? 
  • Why can't I ssh to the server?
    • A duplicate IP problem
  • Why do we have to shut all the servers down for?
    • The AC unit is leaking water all over the floor.
  • Why do we have to shut all the servers down again for?
    • need to redo the electrical wiring
  • Why did problem <X> occur?
    • anomaly 
    • don't know
    • etc
  • What happened to the WAN?
    • Big telco (named <X> having lots of wireless coverage) keeps hosing up their routers or has reoccurring hardware issues
  • What happened to the DB?
    • It was shutdown by mistake.
  • What happened to my test last night?
    • Without warning, desktop group had forced changes and rebooted systems overnight.
  • What happened to my 3 hours of work I forgot to save at 2am when I finally left?
    • Without warning, desktop group had forced changes and rebooted desktop
  • After sending a highly concerned email one Sunday to management and the security team upon recognizing that data within the 3500 emails I received Saturday contained information indicating someone was actively trying to hack one of our systems.. the response was...
    • Security manager: Oh, we were having a company perform a security penetration test.  The enterprise systems manager indicated he didn't think anyone was monitoring.
  • Has anyone made changes to anything (sent to server, network and DB teams)?  The application stopped talking to the DB.  I can access the DB from other applications though.
    • "Nope, no changes here".. 45 minutes later.. "Try it now".  Hm, works fine now so what was wrong.  "Oh, a static route was created that broke things and the person responsible was off today so we didn't know."

to be continued (unfortunately)




No comments:

Post a Comment