Wednesday, October 24, 2012

My IT mistakes: Things I would rather forget

In IT, you can guarantee that there will be bugs, mistakes, glitches and all kinds of other foul things that will occur over time.  Here are a few that I attribute to my own mistakes.  I can only hope that someone else may learn from these.

  1. Did a technology refresh of many of the libraries in an application which performs a form of single-sign-on(SSO) to another application.  The SSO was tested and worked fine but the newer HTTP libraries handled browser detection differently.  The updated SSO system was deployed to production.  The end result was parts of the SSO target application failed to function because of problems with browser detection which had to occur during the SSO process.  
    1. I learned that when dealing with SSO functionality, the target application should be thoroughly exercised as well.
  2. Turned on some extensive logging and ran out of disk space.
    1. I learned to configure logging systems to rotate logs and maintain a fixed maximum disk usage.  I now archive/compress logs off to another filesystem if longer term retention is needed.

to be continued.


IT nightmares

These are things I have experienced that make me shake my head in disbelief.
  • Why did the power go out in the server room?
    • Because someone plugged a fan in.
  • I can't ssh to my current production server, what is going on?
    • Uh, It was accidentally decommissioned
  • Is this decommission ticket correct?
    • Ah yes, I put it in 8 months ago and it indicates it was handled and closed
  •  What happened - why did the network act up?
    • Ah, a network cable came unplugged.
  • What do you mean the manager is having a large scale demo today?  The application hasn't been tested with more than two users.
  • He wants the change in production without being tested? 
  • Why can't I ssh to the server?
    • A duplicate IP problem
  • Why do we have to shut all the servers down for?
    • The AC unit is leaking water all over the floor.
  • Why do we have to shut all the servers down again for?
    • need to redo the electrical wiring
  • Why did problem <X> occur?
    • anomaly 
    • don't know
    • etc
  • What happened to the WAN?
    • Big telco (named <X> having lots of wireless coverage) keeps hosing up their routers or has reoccurring hardware issues
  • What happened to the DB?
    • It was shutdown by mistake.
  • What happened to my test last night?
    • Without warning, desktop group had forced changes and rebooted systems overnight.
  • What happened to my 3 hours of work I forgot to save at 2am when I finally left?
    • Without warning, desktop group had forced changes and rebooted desktop
  • After sending a highly concerned email one Sunday to management and the security team upon recognizing that data within the 3500 emails I received Saturday contained information indicating someone was actively trying to hack one of our systems.. the response was...
    • Security manager: Oh, we were having a company perform a security penetration test.  The enterprise systems manager indicated he didn't think anyone was monitoring.
  • Has anyone made changes to anything (sent to server, network and DB teams)?  The application stopped talking to the DB.  I can access the DB from other applications though.
    • "Nope, no changes here".. 45 minutes later.. "Try it now".  Hm, works fine now so what was wrong.  "Oh, a static route was created that broke things and the person responsible was off today so we didn't know."

to be continued (unfortunately)




Thanks

People, organizations and companies I should give more thanks to:
  • My wife and kids
    • They put up with my long hours, constant inconvenient interruptions and the aggravated mood I end up bringing home more than any of us care for.  I can only thank them for being as patient as they have while I work toward finding a more reasonable environment.
  • Mom and Dad
    •  They put up with a lot with me.  Nice to see how great I turned out though :)
  • Joaquin M, Tim R and their families.
    • Friends who are lights to the world and I wish I could keep in better touch with.
  • Larry and Mari 
    • Miss y'all a bunch; hope that Jesus keeps you and your family close.  Lots of wonderful times at the bible studies which seem so long ago now.
  • My Grandpa Brown
    • For being a wonderful role model for Christian faith &strength and just a great Grandfather in general. 
  • Gregg J, Mark K, Clint, Jeff K and other college friends and their families
    • Many fond memories; wishing it was easier to keep in touch
  • Ray and Al from back home
    • So many good laughs and unforgettable times; wish it was easier to keep in touch with you as well. 
  • Friends I have worked with such as; Rick B, James S, Michael W (x2) and others from CA along with folks from the VCCS such as Violet M, Monica H, Kathy H, Andy C, Jamie W and others. These folks made me laugh, think differently and/or supported me in many ways.   
  • Hanover County Police and EMS
    •  Recently a pretty severe car accident occurred within a stones throw of where I was walking.  As sad as the circumstances were, it was obvious that these folks are awesome at what they do and are a huge benefit to our community.  These folks deserve more thanks than they likely receive and have my respect.  Thanks for being there!
Here are some local businesses which I am thankful for.  In a world where the $ takes precedence over almost everything else these places have not lost sight of their customers.  They have provided us wonderful service and in some cases have almost felt like family at times.
  • Mexico Restaurant - Short Pump
    • http://mexico-restaurant.com/pages/locations.html
    • Great food and wonderful staff.  We go there fairly often and they have watched our kids growing up and it is always a pleasure to be there. 
  • Extra Billy's BBQ - Midlothian
    • http://www.extrabillys.com/ExtraBillysBarBQ2.htm
    • More great food and wonderful staff.  This is another place we have had many family meals at and swapped stories with the staff. 
  • Greek Islands Restaurant - West Broad St
    • http://www.greekislandsva.com/
    • Wonderful people working there.  Feel like part of the family.  My son LOVES their pasta.  The food is great and you get a pretty good sized portion for a very reasonable cost.  Highly recommend stopping by if you get the chance. 
    • UPDATE: Jan 2015; Sadly the owners have decided to retire and closed down.  We will sorely miss both the restaurant and the couple running it.  I do wish them a wonderful retirement though!
  • Chick-fil-A - West Broad St & Short Pump
    • These folks go above and beyond on a regular basis.  Great food, friendly & fast service.  This is the best chick-fil-A I have been to.
  • Applebees - Short Pump
    • Extremely friendly staff, excellent service and food.  They have grown to remember us along with our normal meal choices.   
  • McGeorge Toyota
    • Very courteous and efficient!   They made us, as customers, feel important and they try very hard to meet our needs.  Thanks!
  • Delta Temp
    • http://www.deltatempinc.com/
    • These guys sorted out our messed up geothermal heat pump installation from another company and were more than generous in what they charged.  I wish I had started with them from the beginning.
  • Virginia Urology / Dr. Kramolowsky
    • http://www.uro.com/physicians/eugene-kramolowsky/
    • Couldn't have found a nicer doctor to help me through some kidney stones

Monday, October 15, 2012

F150 brake hoses and shocks

Phew.  After about 1 3/4 weekends, finally got the front & rear shocks and front brake lines replaced on the F150.  With around 10 years of accumulated rust, it was a bear getting some of the bolts/nuts off several shocks and one of the brake hoses.  I only had one nut which required more than a blow torch to remove - it was the driver side brake hose where it connects to the hard line.  This was the line which was nearly burst but a rubber jacket was holding the rust somewhat in place (line bulged about 1/2" more in diameter).  After using the torch on it 6 times or so, I finally got out the pneumatic high speed cutter and cut down the center-line of the connection on the soft-hose side.  Once it cut into the threads a bit, it relieved enough stress on the connection that I was able to back the nut/bolt out (carefully since I had already stripped the head and had to file flat edges back on it).  The downside was the need to cut off the connector on the hard line side, replace the connector and flange the tubing.  I had never flanged a tube before.  Fortunately, the parts store had a tool available and some of the replacement hollow bolts.  I ended up making an extra trip because the replacement hoses didn't come with the required copper washers (which I had not noticed on the ones I took off).  After installing without them and seeing the brake fluid spray about 16 inches - it was quickly obvious what was needed.   I may go back and replace the hard lines as well - I want this to last another 5 years without major issues if possible.  I did more research on flaring hard brake lines and I may go back and do them again.  The directions the parts place gave me worked but I want to make sure it totally safe since me and the family travel using this vehicle.

The shocks in general were not very difficult (other than rust issues) but WD40, a torch, lots of effort and too much time got them off without stripping any components or damaging anything.  The Monroe SensaTrak shocks went on quickly.  My only real concern is the shock tower in the front is rusting through.  Trying to justify a welder to myself and wife - this might be a good start in that.  A small hole in the muffler indicates the adventure might continue.

I had read another blog regarding someone having trouble getting the rear shocks in place because they came compressed with a strap which needs to be cut.  Once cut, it would be difficult to get the shock in place.  I solved it by mounting the bottom of the shock and putting a small length of soft tubing on the top rod/nut.  The tube had enough length to go through the shock tower hole.  Once I cut strap binding the shock, the tube provided enough guidance to the shock as it expanded that one rear went into place without assistance and the other only needed a minor nudge.  If I need to do it again, I would probably use a more semi-rigid tube instead of soft tubing but that is what was handy.

Several laps around the neighborhood went successfully.  Truck doesn't seem to lean quite as severe now and the brakes are not mushy.

Thinking about doing the radiator hoses soon and maybe a new battery.  No indications of water pump issues at this point.  With almost 130k miles on it, I think it is holding up fairly well.