Jeff Billimek’s Journal

As I was about to walk out the door on my way to work this morning, I got a page about a production problem.  There was a conference bridge in-progress so I called-in.  It turns out that there was a problem with some awful queuing problems with the tuxedo server. Jegan and Kumar had been working on it all night.

I worked with Jegan and Kumar to discuss the problem and we weren’t able to nail down the cause of the issue but we did know the symptoms.  Jegan theorized that the problem might be related to a replay TCN file that we dropped into the system last night.  I came up with an idea on how to identify the problem.  We worked through my idea and were able to successfully catch the poison transaction.  It turns out that some of the transactions were being issued in exchange for a dummy ticket (number of all zeros) and we had hundreds of these zero-numbered tickets in the database with about 1,000 coupons each.  Normally a ticket had on average around 3.2 coupons per ticket.

So the result was a transaction message that was so large that it was choking the system.  We worked on getting these tickets cleaned out of the queues andbrainstormed a few ideas for a permanent fix.  I worked from home the rest of the day, babysitting the production environment.

I also ripped an audiobook so that Jen and I can listen to it on our iPods.  The canon rebate thing finally worked out.  Contacting Canon directly was the solution to the problem.  I will be getting the entire rebate that I’m due which is $720!  I’ve never had a problem with rebates in the past and am still quite disappointed with Young America trying to scam me and others.

Jen and I had plans to head up to Katie and Chris’s for dinner.  However Katie contracted the flu and canceled.  So I suggested that Jen and I go out to dinner.  We settled on takeout from a Thai place down the street that we haven’t been to yet.  We’ve talked about it several times because we drive by it all the time.  So I picked up the food and met Jen at home.  She had the spicy basil fried rice with tofu and I had Pad Thai.  It wasn’t as good as the Pad Thai from the other place.  As we ate we watched an episode of the office and then Jen left to go get a pedicure.  I worked on my journal notes as I’m very far behind.  I also worked on some DST testing stuff tonight, read my wired magazine, and went to bed.