Monthly Archive for November, 2003

Nov 1st 2003

Mammoth, day 3

Last night both Buster and Coco slept on my bed. They were both on one side, leaving me about 1/3rd of the width of the bed to myself. I’m actually pretty used to that from whenever a girl sleeps in my bed, that’s usually how it ends up anyway.

I set my iPAQ to wake me up at 5:30am as we wanted to get an early start today. I checked the ftp and all of the photos were successfully uploaded.

I took a nice long hot shower and got dressed. While waiting for Larry I packed my big duffle bag and got my backpack ready for the hike today. It looked as if it snowed most of the night and was still snowing right now. I also had a Myoplex smoothie since we’re not having breakfast today.

Because it is most likely going to be very cold, I decided to dress warm for the trail today. I wore blue-jeans, the tight synthetic stretchy material Nike white shirt, an Abercrombie t-shirt, and my Abercrombie long-sleeved wool sweater.

We left the house around 7am and got to the trailhead around 7:45am. While driving down the little road ascending to the trailhead, Larry mused that the road may be closed due to the snow. Fortunately it wasn’t closed and we arrived at the snow-covered parking lot at the trailhead. We were the only ones there.

The Mosquito Flats (Larry calls it ‘Rock Creek’) trailhead is around 10,000 feet and the car temperature gauge said it was 8 degrees outside. That’s damn cold. The trail started out in the shade due to the position of the rising sun and the mountains.

Despite the cold temperature, I was pretty comfortable except for my fingers (which were sheaved in gloves). The whole area was caked in snow which was quite a treat. The last time we were here, it was summer and there really wasn’t any snow. Fortunately the snow wasn’t deep enough to be a problem so our Lowa hiking boots repelled the snow quite nicely.

Rock Creek trail follows rock creek (as the name implies). The creek goes through several mountain lakes which is really nice. The first lake we came to was mostly frozen. We each picked up a hefty rock and threw them onto the frozen lake - the rocks didn’t fall through the ice.

Eventually the sun rose high enough in the sky to hit the trail. Once we were in the sun, the hike became much more enjoyable. I regained feeling in my fingers and the sun shining against the ice-kissed foliage was a sight to behold. Larry brought along the tripod again today and took full advantage of it. We stopped for quite a while at several different spots.

One place we stopped was a large clump of shrubs with ice & frost on the leaves. The way the sun was hitting the leaves gave a sparkling menagerie of lights which was begging to be macro photographed. Most of the creek was frozen with a few patches of flowing water. There were also interesting icicles and frozen spots that proved to be great photographic subjects.

For most of the trip I had my Sigma 15mm fisheye lens on my camera and took many wide-angle photos of the snowy meadows and snow-covered mountains in the background. I had a great time being a little creative with the fisheye lens, photographing the landscape with a large object in the foreground (like a rock or stump or something). Because we got an early start, the sky was clear.

We eventually turned around when we saw ominous-looking clouds building behind us in the valley below. On our way back we passed two other groups of people.

It was really cool to go on this trail early in the morning and the snow was an added bonus. I think we both got a nice bounty of some good photos, between my wide-angle landscapes and Larry’s tripod-assisted macros.

When we got back to town, we agreed to go to Roberto’s again for the third day in a row. It’s just so good that it’s not bad to have it so often. Like before, we both had the carne asada burrito.

Once back at the house, we set a 2pm departure deadline and packed everything up. We ended up leaving at 2:10pm. On our way out of town we stopped at the video store to drop off the movie.

From Mammoth Lakes, we headed south on highway 395. We descended and went through Bishop, then Big Pine, then Lone Pine. Lone Pine is the staging area for Mt. Whitney. The whole time we followed along the eastern sierra mountain range. Eventually we came out of that and into a desert area. The sunset was really nice against the flat desert area.

For most of the long car ride home I was thinking about blood clots. I’ve been reading a lot about people who sit on airplanes a long time who develop blood clots in their legs and die. I hope that doesn’t happen to me!

When we got to San Diego, Amy had the pumpkins lit on the front porch. Her and Rachel did an excellent job carving the pumpkins with designs from ‘winnie the pooh‘ characters.

I fixed myself some leftover spaghetti and worked on Larry’s computer. I downloaded and installed Ad-Aware. I’ve never really used it before, but it was simple. I simply updated the ‘definitions’ and ran a full system scan. It found over 160 ‘objects’ with infections. I had it clean everything up and rebooted.

I also set up an FTP-link from the WinXP box to Larry’s iBook and moved all of the photos from the laptop to the PC.

Nov 2nd 2003

Balboa Park

I slept in the very comfortable guest room that Larry and Amy have set up with the fluffy full-sized bed and Egyptian cotton sheets. I ended up oversleeping until around 7am PST.

I woke up and fixed myself a smoothie and puttered around for a while. Tucker and Lily were watching a video on ‘big trucks’ featuring monster trucks, garbage trucks, fire engines, etc.

Around 9am Larry and Amy decided that we should head down to the Balboa Park area to visit the museums and have lunch at Prado. I packed up my bag and changed into my ‘airplane clothes’. We loaded the suburban with the stuff and the three kids.

I’ve never been to Balboa park so I didn’t know what to expect. Of course anything we do in San Diego turns out to be fun so I wasn’t worried. While we were driving into town, we passed a nice stretch of highway with a grassy trail along the side. I’ve always noticed this before but asked about the area. It turns out that this area is the San Diego Zoo and Balboa Park.

It was still early (around 9:30am) so not much was open. We parked and walked around among the buildings. Tucker was obsessed with two little seeds he found on the ground. Larry explained that Tucker loves to grab onto tiny objects and then it’s a big tragedy whenever he loses them.

We swung by a nice botanical garden area with a pond outside. Inside a large building was the fauna area.

After that we went to the science museum (around $7 per adult) and spent about an hour in there. I think Tucker and Lily had a nice time with all of the scientific stuff inside.

At 11am we walked over to Prado for lunch. It was a nice restaurant with attractive young blonde girls greeting and seating people. I had the special which was the apple-wood smoked bacon risotto with bay scallops.

After lunch, they dropped me off at the airport. It wasn’t that crowded and I had no problems checking my bag and getting listed on the flight. I got to sit up front in business class (seat 5G).

The flight was very enjoyable. I flipped between ‘The League of Extraordinary Gentlemen‘ and ‘American Wedding‘. Even though I’ve seen American Wedding already, there are some great scenes to see again (notably, the bachelor party scene).

I think I overindulged in alcohol on the flight. I can’t remember how many crown & cokes I had but it was around four or five. In the future, I don’t think I’ll have that much.

When I got home, I uploaded the rest of my photos and spent way too much time editing them in Photoshop. I didn’t get to bed until 4am. I set my alarm for 8am so I could at least get four hours of sleep. I know that I’m going to pay for it tomorrow when I’m feeling like crap.

I created a photo album for the Mammoth trip and broke it down into three sub-albums - one for every trail we went on.

Nov 4th 2003

All work and no play makes Jeff a dull boy.

I’ve been so busy that I’m writing this journal entry five days late from memory.

We were told that all code changes for production are to be in by tomorrow.

This afternoon Jegan came by and told me that he finished writing a helper class, ‘RPWriter’ which will assist in the creation of special log files. A series of environment variables control weather or not the logging is enabled, how often files are created, and other things relating to the filename.

I worked on incorporating this functionality into the four translators. I worked on one first and went through a few coding/testing iterations. When I was pretty happy with the results I shared it with Casey. When we were both looking at it, we discovered a problem with the timestamp on the filenames. We made some tweaks to include a date stamp in the filenames as well.

It took longer than expected to finish the work. Casey was also working on a last-minute change to one of the point-of-sale services too. Apparently there was a requirements gap and he had to make some very last-minute changes. We both had our code changes wrapped up around 10pm.

Because we had a change to the ‘common library, I had to do a ‘top-down’ build of everything. It was definitely going to be a long night. Before I left the office, I kicked off the first part of the build and drove home.


(I snapped a photo with my camera as I rode down the elevator at 10pm)

As soon as I got home, I continued working. I ran into a snag. The build server process wasn’t distributing the built-binaries correctly. There was nothing I could do until someone from release engineering could take a look at it. I finally collapsed in bed around 3:30am. I didn’t get a chance to have dinner.

Nov 5th 2003

the day from hell

I woke up at 6:30 and immediately resumed work. Bob O’Brien from release engineering fixed the distribution problem. The build server couldn’t find the Tivoli endpoint on the network due to some DNS changes yesterday.

I worked straight until about 8:30 at which time I took a shower and drove into work.

It was pretty anti-climatic to finally get all of the code into system test as early as possible only to find out that they couldn’t begin testing yet. There were yet more DB2 errors.

The issues didn’t get corrected until the afternoon. Since I only had a few hours of sleep, I was pretty out of it today. I did sit on the integration-test bridge calls and assisted where I could.

Jegan offered the activate last night’s changes in the integration environment, so I finally left around 4:30pm.

It was a dreary rainy day. As soon as I got on I-285, I hit gridlocked traffic. What is normally a 30-minute commute turned into a 2.5-hour commute. At one point, it took me thirty-minutes to go 0.5 miles - on the interstate highway.

This was definitely an exercise in patience and self-discipline. I don’t think I did too well.

I went to bed again today without having dinner.

Nov 6th 2003

all work and no play makes jeff a dull boy

While yesterday was supposed to be the absolute last day for code changes, the ‘MTR’ group in my team had some critical bugs to fix.

I had a decent amount of sleep last night so I was a little sharper today.

Because of the late code-fixes, I was at work until about 10pm again doing all of the builds and getting the code into system test. We decided to wait until tomorrow morning to activate the code in integration.

I had some snacks for dinner tonight before going to bed.

Nov 7th 2003

photo friday: ‘Downtown’

Today’s theme is ‘Downtown


(click on the photo for a larger version)

Nov 7th 2003

aLL WoRK aND No PLaY MaKeS JeFF a DuLL BoY

I’ve been so busy lately that I’ve put off doing my 2004 benefits stuff. It’s due in by the 12th. Last year our medical and dental benefits used to be free. This year they started charging us, due to the rising health-care costs. Now for next year, my medical premiums are going to be going up 276%. I’m not too happy about that.

Around 10am I started work on activating the code in integration while the MTR team was working on yet another critical code fix.

Today is Scott’s 30th birthday. A large group of people were taking him out to lunch. I was invited but couldn’t go because I was in the middle of the activation. Around 1pm Alex (who was busy working on the MTR changes) and I ran to Wendy’s to pick up some lunch. We brought it back so we could continue working while we ate.

This afternoon we had a large ‘PRB’ meeting with all of the managers from the various groups with a stake in the project. Alonzo (my director) went over all of the outstanding issues and the current status.


(Everyone intent on meeting our delivery date)

At one point the meeting got pretty intense when there was discussion around the weekend schedule. The System Test team wanted to wait until 6pm Saturday to test today’s fixes and tomorrow’s fixes. Carole & others wanted to run a batch of tests tonight and then another batch tomorrow. The reasoning is that if we discover a problem with the fixes tonight we’ll have more time to fix them than if we wait to discover them tomorrow night. Finally we decided to test twice.

Again I worked late. The MTR team was busy making some last-minute code fixes. Some of us joke that they do ‘extreme programming’ by cramming five people into a cube until the bug if fixed.


(extreme programming?)

Once they completed their changes, I began building all of the code. When I was about half-way through it, Sameer came by and said that they discovered another problem which promoted me to begin the build over again. This cost me about an hour or so.

During one of builds, I watched Sameer and Rodney play a game of chess while we waited. It was a pretty good game. They were about evenly matched and it developed nicely. Unfortunately I left around 9pm so I didn’t see the conclusion of the game.

When I got home, I continued working. I finished the build and got it into system test. Once the system test team validated the changes, I activated them in integration. I finished up around midnight.

Nov 8th 2003

aLL wOrk anD no PLAY maKEs JEff a dULl Boy

As soon as I woke up at 9:30am I dialed into work. I checked the health of integration test and to my horror I saw that we had tons of core files. The 5GB log file system on each of the six nodes was at 100% due to all of the core files being created.

It was readily apparent that we had a major problem. I paged Carole and John with an update. I also paged Rodney. Rodney and Sameer were in the office this morning. I explained the situation to them and started sending over core files for them to look at. I also sent the stack trace which wasn’t very helpful.

During this time I was in constant contact with Rodney and Sameer. Eventually they thought they found the ticket number of one of the tickets causing the core dumps. They requested that ODE send them the transaction for that ticket number.

Unfortunately ODE could not find that ticket in their error queue. I think Rodney took this to mean that there were _NO_ tickets at all in their error queue. He then came to the conclusion that the tickets were eventually making it through the system when retired enough times. This would mean that the problem wasn’t always reproducible - meaning this could be an intransient memory issue.

Even though they didn’t know what was causing the problem, Rodney and Sameer requested that I rebuild one of the components to see if re-linking would fix the problem. I agreed but also suggested that I temporarily enable logging so we can capture the message causing one of the next core dumps. They agreed to this and I set out to do both tasks.

Around this time Mehdi from ODE sent us an email with the 976k logfile from the error queue. It looks like Sameer and Rodney misidentified the ticket number causing the problem. I suggested that they take a look at one of the tickets in the file Mehdi sent.

About an hour later I got a page from Rodney. They finally found the root cause of the problem causing the core dumps. The code fixes were checked in and they requested that I build the two components as soon as possible.

I labeled, merged, and kicked off the build of the code. It didn’t look like I was going to have it ready by 3pm.

Once 3pm rolled around I called into the bridge line for our scheduled 3pm status conference call. I gave everyone an update and then listened to the managers go over the schedule for the rest of the day. The game plan is to wait for me to activate the current build in system test. They are going to do a ‘comprehensive’ test in system test so we don’t have the snafu like we had last night. We agreed to have another bridge call at 7pm tonight.

While I was working today I watched ‘The Shining‘ and then after that I watched ‘Rich Girls‘ on MTV.

Around 4pm I had the code deployed and activated in the system test environment. While I waited, I paged Chris A. and Tim B. from mid-tier engineering to have the binaries deployed to the six integration servers. It wasn’t until around 6:30pm that they validated the results in system test. I paged John to see about activating in integration. Have gave me the go-ahead.

It’s a tedious, time-consuming process to activate code in integration. I have to do it one node at a time. I start with the last node and work my way up. By the time I joined the bridge call at 7pm I had only completed the first node. I continued to work while on the conference call.

Finally around 8pm I finished activating and they turned the ticket flow back on. I monitored the system for the next hour and didn’t see any more problems. There were no core dumps and everything seemed smooth.

I paged Carole and John with an update. Before logging off for the night, I entered my time for this week into PlanView. This was by far my longest week of my career. I logged 73.5 hours.

Nov 11th 2003

2003 Developers Conference

Instead of going in to work this morning, I headed over to the Georgia International Convention Center near the airport off of Camp Creek parkway. For a few weeks now there have been advertisements in the lobby and around the building about a ‘Developers Conference’ for all ‘developer’ family of employees within the organization. They have been making a pretty big deal about this and have been presenting the banners and advertisements with a comic-book style of artwork.

When I arrived around 7:45am, I got in line for registration. Because all of the people there were developers and managers, I recognized many people. In fact, Casey was next to me in the registration line.

While registering, they gave me a bag with paperwork including a brochure booklet that was presented much like a comic book. I also got a 32-meg flash memory keychain device for filling out a survey.

As soon as I got through registration I went into the main conference room for the 10am keynote speech. Casey, Brian, and Rodney saved a seat for me. The keynote speech was interesting.

First to speak was Mike Childress, a senior vice-president. He talked about the IT industry in general as well as shifting our focus more to ‘engineering’ style development instead of ‘hacking’ style development. In order to achieve this, we use a standard set of tools and methodologies.

After Mike spoke, the second speaker was Carey Stewart (director of Architecture). Carey did a role-playing thing describing an evil secret force that has been infiltrating our ranks. He unveiled this threat by pulling a giant sheet off of a large stage prop which was a gigantic robotic spider. The demonstration continued with a developer ‘firefighter’ to try to eliminate the bug with a fire extinguisher. Unfortunately it didn’t phase the spider. Next, a team of new firefighter developers armed with the new Windows XP suite of products used their high-tech tools to kill the spider. Instead of using a fire extinguisher, they used some sort of flash-bang device. There was a loud explosion and lots of light and confetti from the ceiling. I wonder who is footing the bill for all of this? Carey also talked about something very interesting. He said that for windows development we are going to be moving to .NET and C#. This was very interesting to hear because I sent him an email about 1.5 years ago asking if there was any plan to move to .NET and C#. At the time, his response was that we had no intentions of moving in that direction. I guess things change.

After the speeches, I headed over to the ‘book room’ where Borders set up some booths. For attending the conference, Developers get a free copy of ‘XML Programming Bible‘ as well a free book of our choice. I decided to get ‘.NET Patterns‘.

I walked around to some of the other rooms where different vendors had booths set up demonstrating their products. There wasn’t a whole lot new for me to learn at the conference.

Around 11am I went back to the office and worked for a couple of hours before going home. Craig planned on arriving today around 2pm so I wanted to be home before he got there.

When Craig finally showed up, we went to Fuddruckers and had a late lunch. We caught up over lunch and then went back to my place. I helped Craig unpack his car and get settled in. He brought his old computer that he’s going to be giving to his cousin Jeff. I helped him rebuild the OS and get it cleaned up. He also brought all 25 Harry Potter book 5 audio CD’s as well as the Lord of the Rings trilogy.

Around 7pm we left my apartment and headed out to the movie theater. We saw ‘Matrix: Reloaded‘. Unlike some of the reviews, I thought it was a good movie. I thought there was indeed some closure at the end. Some of the parts were pretty slow but I still liked it.

After the movie we stopped at the package store and bought some liquor. Craig got some of those Bacardi single-drink things and I got some vodka and Red Bull.

Back at my place we played Warlords 4 all night until around 4am before going to bed.

Nov 12th 2003

wasting away being a warlord

The original plan was to wake up early and go hiking around Kennesaw Mountain this morning. Reality was that we ended up sleeping in until around 11am.

When I woke up, I found Craig playing Warlords in the living room. I joined him (since it’s a hotseat game) and before we knew it, it was late. We planned on going to Athens today to visit Jenn but because of the time, we didn’t want to get caught in the afternoon traffic. We decided to forego Athens.

Around 6pm we headed out for dinner. I’ve been itching to go to Houston’s for a long time now because I haven’t been able to - and I really like Houston’s. We had a leisurely and enjoyable dinner.

After dinner Craig really wanted to go to a gentleman’s club but I wasn’t up for that.

Nov 13th 2003

mardi gras

I woke up around 6am today and went into work, leaving Craig asleep. I worked for half a day. Originally I was going to come into work this afternoon because the original plan was to take revenue pipeline to production this evening. There was a problem yesterday which caused us to push our “go-live” date to tomorrow.

When I got back home around noon, Craig was awake playing on the computer. We left around 12:30 for lunch. I took Craig to Smokey Bones off Ashford-Dunwoody road. I had a full rack of baby-back ribs. It was truly delicious! During lunch we watched the Kasparov vs. X3D Fritz chess match on ESPN2. I was surprised to see ESPN covering a chess match.

Back home I watched a little more of the chess tournament while Craig played on the computer.

Around 3:30pm we left and drove to Phipps Plaza mall to see the special director’s cut of Alien. It wasn’t too exciting. I’ve seen it before but it was nice to see it on the big screen. I like Alien’s better.

After the movie we walked around the mall for a while before going back home.

Back home we played Warlords a little (I think I’m seeing a pattern here). We used to play an older incarnation of this game as well as ‘Heroes of Might and Magic‘ back when we were in high school. It was a lot of fun to play in large groups because of the ‘hotseat’ feature in which players trade-off taking turns.

We were going to head out for dinner around 8pm but I convinced Craig that we should delay because Survivor comes on at 8pm and my VCR broke so I can’t tape it. We watched Survivor and then left for dinner.

For dinner we went to Copeland’s around 9pm. I had a large salad with honey bacon mustard dressing. It was pretty good.

Back home Craig packed up his bags and I took a power nap.

Around midnight Craig and I left for a gentleman’s club called Mardi Gras that’s actually pretty close to where I live. When we got there, we had to valet park. There was a $10 cover per person. We found a table and ordered some drinks. Craig got a beer and I got a Red Bull & Vodka, the cost of both together was $19. Mardi Gras is a full-nudity club with a few stages. Craig went up to the main stage to tip a well-endowed dancer. When he got back he complained that unlike San Antonio, the dancers here don’t do a special private dance for you when tipping them. After observing another table get a lap dance, Craig commented that there doesn’t seem to be much contact - possibly due to them being totally nude. Craig theorized that if they were wearing something the lap dances may contain a little more contact.

The dancer that Craig tipped from earlier came by and Craig ordered a lap dance. Her routine was pretty much like all of the others - very scripted and predictable. I think she had implants.

We didn’t stick around for much longer after that. Craig was pretty disappointed in the ‘quality’ of the place but I can’t really judge since I don’t have enough experience with that sort of thing. It may just be a difference between Georgia and Texas.

Nov 14th 2003

photo friday: ‘Work’

Today’s theme is ”Work


(click on the photo for a larger version)

Nov 14th 2003

baby shower

I planned on going into work late today because the scheduled activation of the Revenue Pipeline application isn’t scheduled to begin until 7pm.

Craig needed to leave early because he’s driving 11-hour to Chicago today. I helped him get his stuff together and he left around 6:30am.

After Craig left, I set my alarm for 10am and went back to sleep. I woke up at 11 and got ready for work. While I was getting things together I got a page from Karen. Karen is my vice-president’s administrative assistant. She paged me asking me to go to Brent’s office. This presented a challenge because I wasn’t even at work yet. I paged her back explaining that I’m coming in late today due to the 7pm load.

I checked my email and noticed a nice email from ‘Linda’ regarding my website:

“All I can say is, You are very ,very gifted.I really enjoyed looking at all your photos.Thank you for sharing them with all of us”

It’s nice to get a compliment like that!

I arrived at work around 11am and settled in. Today at 11:30am in the 10-East conference room we are having a baby shower for Scott and his wife, Krissy. Everyone brought in food and we all pitched in to buy a gift.

I had my backpack with me, and so I had my camera. I slapped on the 15mm fisheye and took it along with me. During the shower I took some photos. It was tricky lightning conditions because pretty much everyone was back-lit since the bright sun was shining through the windows along the wall. This made many photos of people with their backs against the window very tricky. When it came time for some of the photos of Scott and Krissy opening stuff, I really could have used something a little tighter than the 15mm. I tried to make the best with what I had though.

I stopped by Brent’s office after the party and he gave me instructions to carry out some covert activities.

As soon as the baby shower was over, I began working tweaking the script I wrote to archive log files. I decided to change the way in which files are identified to move to the archive server. Previously I was only comparing the current and previous hour against the hour timestamp in the filename. Now I’m making an embedded perl call to compute a custom datestamp n-number of hours back. There is a snag however. Our ’standard’ HPUX 11.00 installation comes only with an ancient version of perl (4.0). This prevented me from using any of the packaged solutions found online. Fortunately Jegan helped me come up with a command that worked within the confined of perl 4.0. Once I achieved this functionality it was trivial to expand it to the ‘rolloff’ capabilities of the script to automatically delete archived logfiles over x-number of days old.

The production load of Revenue Pipeline 1.2 was going ahead of schedule today and it looked as if the piece Jegan and I need to work on would happen earlier than the scheduled time of 7pm. Indeed, we managed to begin at 4pm. Jegan had everything pretty much under control because he and I have been doing activations in integration so much we can pretty much do it blind now.

While Jegan worked on the activation, I continued to work on and polish up the cron script. I did some extensive testing and then loaded the new version into integration.

The activation in production went smoothly and there were no problems. The ticket flow won’t be turned on until tomorrow morning at 7am so most people have cleared out of the office by now. I was still working and noticed a nice sunset which I snapped a photo of:

I worked until around 7pm. When I got home I basically collapsed and went to bed.

Nov 15th 2003

go live

I woke up around 5:30am and got ready for work. It was unusual for me to drive into work while it’s dark outside. When I arrived, Carole was already in her office. I let her know that I had arrived and settled into my desk.

I saw that the sun was beginning to rise, and the cloud cover was just right to result in a spectacular sunrise. I grabbed my camera and headed over to the 11-East conference room and closed to door (to block out and glare of lights against the windows). I snapped a few photos of the sunrise with both my 15mm fisheye and 28-135mm rectangular lens.

Back at my desk, I hopped on the 6:30am bridge call. Jeff Chambers was running the call and quite a few other people were online too. Originally we were supposed to turn on the ticket flow at 7am, but there was some sort of hawk/rendezvous problem last night that they needed to resolve.

Jegan came in while we were waiting. I checked the health of my hourly logging script in integration and everything seemed fine, so I installed it on all six servers in production.

We didn’t actually get the ticket flow turned on until about 8am. As soon as the tickets started to flow through the system, ODE reposted some errors from DB2. At the same time, we dropped a core file on one of our tuxedo nodes. The output translator in the north was the culprit. I ftped the core file to the development server and loaded it up in the debugger.

Analysis of the stack trace didn’t reveal anything spectacular but I was able to identify the ticket number of the ticket that caused the error. I emailed the stack trace to Rodney who was at home but on the conference call.

While we were working on this, Vann from middleware came up to the 11th floor and wanted to know if any of us wanted breakfast. He went to McDonalds and got me a sausage & egg biscuit. It was pretty good.

Back on the bridge call, Rodney said that the problem we are seeing is exactly the same problem he saw and fixed before. He suggested that the configuration was wrong. I verified the configuration across all six nodes as well as comparing them with integration and saw no differences. I emailed the results to Rodney and over the bridge call we discussed this and theories about what the problem may be.

Carole came by a short while later and suggested to me that we keep the detailed discussion off the call because there is a large audience listening - including our vice president. She told me that she also paged Rodney with the same thing. Her words made sense and I see the wisdom in what she said. She wasn’t telling us to cover up the fact that we were looking at a problem, but rather to solve it in private - that’s what all of the other groups are doing.

I extracted the ticket from one of the log files but I was unable to reproduce the error in the development environment. Around this time Rodney declared that he found the source of the problem. The Electronic Ticketing group was sending duplicate messages with duplicate keys which were causing our process to die. This type of condition isn’t ’supposed’ to happen, but it did. Later we found out that it was an unintended side-effect of the temporary DB2 problem we had. Sameer called me later and we chatted about the problem. Both he and I agree that our app should never drop a core in a situation like that. At the very least we should throw an exception. Sameer said that he is going to code that fix on Monday.

That was the only problem we had today with the tuxedo applications. About thirty minutes after we worked through the DB2 and core dump problems, we saw severe performance degradation. Research revealed that one of the oracle servers were under massive disk IO. We tracked this down to a query which was doing a full table scan every transaction. In integration we had the same type of problem but corrected it by creating an index.

It took some time (dealing with the oracle DBA) but we finally decided that the index was missing from production. We were told that it would take some time to correct so we turned off the ticket flow into the application.

Casey had also come in the office this morning. While we waited, Carole came by and took lunch orders. There were about ten of us in the office at the time and we decided to order pizzas from Johnny’s. I went with Casey to pick up the five pizzas we ordered. We ate lunch around 1pm in 11East.

The DBA got the index created and we turned the ticket flow back on. The performance problem we observed earlier went away.

The rest of the day was pretty smooth with only a few more bumps. As the afternoon wore on, I saw a sunset out of the west windows and captured it with my camera. It’s strange that I saw both the sunrise and sunset today from the 11th floor.

I finally left work around 6pm. This was much later than I originally planned. I made plans to meet Jenn after she gets off work today (7pm) for dinner. I raced home so I could change and the immediately headed back out to Lenox mall.

I met Jenn in the Christmas-decorated store where she works. She looked great! She was wearing a short black skirt with thigh-high black boots and a powder-blue turtleneck sweater. To top it off, she was wearing cool pointy-frame black glasses which looked really good on her.

We decided to walk to a restaurant nearby called Milan. We sat outside and had tappas. One of the things we had which was quite memorable was bacon-wrapped dates.

Nov 18th 2003

1&1

Yesterday I discovered an offer for three years of free web hosting by a company called 1&1. The deal is 500 megs of storage and 5GB of bandwidth/month free for three years. It looks too good to be true but everything I read about it (from skeptical people on the net) indicated that it’s the real deal. I, of course, signed up and registered a new domain: leased-territories.com.

The 1and1.com ‘control panel’ interface isn’t that easy to use and they don’t give many DNS options when dealing with your domain, but it’s free. The space should be sufficient enough for me to mirror my entire site over there (once I clean up my photo gallery).

I brainstormed today about how I could achieve mirroring billimek.com on leased-territories.com and came up with the following ideas:

1) The phpBB and movabletype MySQL databases will need to be dumped, copied, and loaded into the mysql server at 1&1.
2) The entire html directory tree on billimek.com will need to be copied over to 1&1. A great way to achieve this is to use the ‘rsync‘ utility. It does an intelligent synchronization, utilizing compression during transport and only updating the parts of files that have actually changed.
3) Because the two sites will have different domain-names, there will be considerable challenges in configuration files, database rows, etc containing a hard-coded billimek.com domain name for the site which will break things when being run under a different domain name.
4) 1&1 only runs a POP-based mail server which is unacceptable to me. IMAP is far superior so currently the mail server functionality must remain on my linux box at home.

This is an exciting little project for me because I’m going to be learning a lot of new things as well as getting an opportunity to practice migrating my site to a hosting provider - something I’m probably going to be forced to do when I move because I’ll probably not be able to retain a static IP address.

It was a pretty busy day at work day. As soon as I got in the office, I jumped on the bridge call and we reviewed the status of revenue pipeline 1.2 production. Things are going pretty good, at least from the tuxedo server application standpoint.

Immediately following this was an hour-long meeting with all of the managers of the project. It was a short meeting today - we just went over the status of the system and the major issues.

Instead of going to lunch today I had a lunch-meeting in 12-East. We discussed the reservations system sync processing. It looks like we have around 13 million transactions to run through the system so that we get synchronized with the res ‘cache’.

I spoke with Larry who invited me out to San Diego over thanksgiving. I checked the flights and it actually looks doable to go out there early Thursday morning and return sometime on Saturday. I also checked flights to/from Austin around the same time and they didn’t look nearly as good. I may spend thanksgiving in San Diego.

My lease is up at the end of this month. Six months ago I renewed my lease with the intention of buying a house between then and now. I still haven’t gotten around to doing that. I now have a choice. I go month-to-month and pay a whole lot more per month, or I renew for six months and pay only a little more per month. I’ll probably renew for another six months and start getting serious bout home buying.

This evening I upgraded my photo gallery software to the latest version (1.4.1-RC3). The new version is pretty cool because they have many different ’skins’ one can choose from.

One of my personal tasks has been to prune my photo gallery. Until a few months ago, I posted monster-sized photos to the albums which eats up a lot of space and bandwidth. I thought the built-in resize option in software would allow me to resize all of my original-sized photos, but that didn’t work. I then looked into using some command-line utilities to resize but that wasn’t working either (it was too tedious and didn’t work correctly). I finally found a great tool called ‘myresize‘ which is a plugin for gallery which will automatically resize all of your photos to a specified size. I ran this and it worked like a charm. My disk-space usage went from 650MB to 280MB after I resized everything to 800×600.

After resizing, I went through each album and pruned out photos that I didn’t think were necessary. I’ve used a lot of discipline with the later albums so there wasn’t much work to do with them, but some of the older ones had way-too-many photos.

As I was going through my albums, starting with the oldest ones first, I noticed an interesting trend. The further back I went, the worse the quality of the photos. Of course, most of the stuff before 2002 was film (scanned). However I think the quality of the camera (as I’ve upgraded my digital camera a few times since 2002) could have something to do with it too.

Nov 19th 2003

phpBB and movabletype

I’ve been listening to the Harry Potter and the Order of the Phoenix audio CD on my drive to and from work. It’s such a treat because Jim Dale does a great job narrating the different voices. My commute to and from work are quite enjoyable now.

Today was a fairly slow day at work. Things are moving along in production. We had to turn off the ticket flow for several hours so the DB2 DBAs could do reorgs on some of the tables to speed things up.

I upgraded phpBB to the latest version (2.0.6). After upgrading it, I did some more work on the synchronize script to mirror everything from billimek.com to leased-territories.com (hosted on 1&1).

With a little effort, I was able to mirror the ‘netheril‘ BBS on the new host. I even have the script set up to extract the database from my local mysql databse, upload it to leased-territories.com, massage the mysql dumped file to replace the domain name, and re-insert it into the remote database.

Also this evening I upgraded movabletype to the latest version (2.64).

I went to bed early because I was up so late the night before.

Nov 20th 2003

find /html -type file -name ‘*.php’ | xargs sed ….

In the morning revenue pipeline core team meeting, we discovered that there was an issue with a lot of missing coupons in the RPAD database. It turned out that the extract wasn’t working correctly and many coupons were being skipped.

I made a lot more progress with my domain mirroring project. I managed to mirror everything from billimek.com to leased-territories.com. I spent most of my time polishing up the sync scripts to get everything working correctly.

There are still a few snags I need to work out though. While I can ’sed-out’ the hard-coded names in the database, I’m still at a loss at how to handle the domain being hard-coded in individual files. It turns out that there are many such occurrences in various files. While doing a ‘rebuild’ from movable type, most of the hard-coded domain names get fixed based on the configuration value in the database. Unfortunately all of the ‘image’ pages remained the same as they are not tied to the database and are not ‘rebuilt’.

This poses an interesting challenge. There are over 370 such files in jeff/images containing a hard-coded domain name. If I want to change domains, I’m going to have to sed those files as well. Maybe a recursive sed using ‘find /html -type file -name ‘*.php’ | xargs sed ….’ would do the trick? I’ll have to investigate this further.

Nov 21st 2003

photo friday: ‘Gatherings’

Today’s theme is ‘Gatherings


(click on the photo for a larger version)

Nov 21st 2003

mysqldump

I came to the revelation that I have a problem if I want to move form the 1&1 mysql database to my local server. Locally phpbb is in its own database and movable type is another database. On 1&1 I have only one mysql database to play with. So when I do a mysqldump from that database on 1&1, I have a file with both sets of tables. I can’t cleanly split this to two separate files to use on the two separate databases on turing.

I finally came up with an answer, in involved a weird hack using the following commands:

mysqlshow -h${DB_HOST} -u${DB_USER} -p${DB_PASS} ${DB_NAME} phpbb_\* | awk ‘{print $2}’ | grep phpbb

and

mysqlshow -h${DB_HOST} -u${DB_USER} -p${DB_PASS} ${DB_NAME} mt_\* | awk ‘{print $2}’ | grep mt

These commands will return a list of only the tables I want. I can then use them embedded in the mysqldump commands in my ‘backup_database.sh’ script on the 1&1 host:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/bin/bash
 
# DB host
DB_HOST="xxxxx.perfora.net"
 
# DB name
DB_NAME="dbxxxxxxxx"
 
# DB user
DB_USER="dboxxxxxxxx"
 
# DP password
DB_PASS="xxxxxxxxxx"
 
# phpbb
mysqldump --opt -h${DB_HOST} -u${DB_USER} -p${DB_PASS} ${DB_NAME} `mysqlshow -h${DB_HOST} -u${DB_USER} -p${DB_PASS} ${DB_NAME} phpbb_\* | awk '{print $2}' | grep phpbb` > ${HOME}/dbxxxxxxxx_phpbb.sql
 
# mt
mysqldump --opt -h${DB_HOST} -u${DB_USER} -p${DB_PASS} ${DB_NAME} `mysqlshow -h${DB_HOST} -u${DB_USER} -p${DB_PASS} ${DB_NAME} mt_\* | awk '{print $2}' | grep mt` > ${HOME}/dbxxxxxxxx_mt.sql
 
# fix permissions
chmod go-rwx ${HOME}/dbxxxxxxxx_*.sql

My ‘restore_database.sh’ script on 1&1 looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/bin/bash
 
# database server
DB_SERVER="xxxx.perfora.net"
 
# database name
DB_NAME="dbxxxxxxxx"
 
# database username
DB_USER="dboxxxxxxxx"
 
# database password
DB_PASSWORD="xxxxxxxx"
 
# from
FROM="http:\/\/billimek\.com"
 
# to
TO="http:\/\/leased-territories\.com"
 
# phpbb_db
#gunzip -f phpbb_db.txt.gz
#sed -e s/'cookie_domain','billimek\.com'/'cookie_domain','leased-territories\.com'/g -e s/'server_name','billimek\.com'/'server_name','leased-territories\.com'/g phpbb_db.txt > phpbb_db.txt.new
#mv phpbb_db.txt.new phpbb_db.txt
#mysql -h${DB_SERVER} -u${DB_USER} -p${DB_PASSWORD} ${DB_NAME} < phpbb_db.txt.new
mysql -h${DB_SERVER} -u${DB_USER} -p${DB_PASSWORD} ${DB_NAME} < phpbb_db.txt
#gzip -f phpbb_db.txt
#rm -f phpbb_db.txt.new
 
# mt_db
#gunzip -f mt_db.txt.gz
#sed -e s/${FROM}/"${TO}"/g -e s/"\/var\/www"/"\/kunden\/homepages\/14\/dxxxxxxxx\/htdocs"/g mt_db.txt > mt_db.txt.new
sed -e s/"\/var\/www"/"\/kunden\/homepages\/14\/dxxxxxxxx\/htdocs"/g mt_db.txt > mt_db.txt.new
#mv mt_db.txt.new mt_db.txt
mysql -h${DB_SERVER} -u${DB_USER} -p${DB_PASSWORD} ${DB_NAME} < mt_db.txt.new
#gzip -f mt_db.txt
rm -f mt_db.txt.new
 
# fix permissions
chmod go-rwx ${HOME}/*_db.txt*

I’m pretty happy with the results. The sync scripts are almost done.

I did more pruning of my photo gallery today. I removed all intermediate images (640×480) so that I only have the thumbnails and the ‘original’ 800×600 images. This reduced the disk-footprint of my entire album tree to 165MB. Not bad considering how large it was before.

I ended up staying at work late today. Carole came by and told me that they were assigning my pager as the primary ‘on-call’ duty page for revenue pipeline until further notice. This means that I should be the first to get paged for problems. Lovely.

Throughout the day Casey was looking at another potential problem with HRRC. I helped him out by extracting tickets from our DTMO logging system. It’s worked out really well that we implemented that.

At first it looked as if we were overstating revenue by a great deal due to this bug. We were prepared to make an emergency code deployment to production to fix the problem, but as they looked closer at it, they determined that it only affected 2% of 1% of all tickets. Based on this, management decided to hold off on implementing the change until after the Thanksgiving code freeze.

After work I drove up to Athens to visit Jenn. We watched ‘The Bourne Identity‘.

Nov 23rd 2003

DNS pointing

It’s been an unproductive weekend and I lament that I didn’t spend much time outside since both Saturday and Sunday were beautiful. I’m sure I’ll make up for it though.

I managed to wrap up my synchronize scripts so I’m pretty happy with that. Actually they were pretty much done, but I didn’t finish up the last parts because I knew I already ’solved’ them.

Instead, I worried about transferring the DNS registration of billimek.com.

First, ’sync-to.sh’ is used to synchronize all of my local host’s content to the stuff on 1and1.com:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/bin/bash
 
# local directory 
LDIR="/var/www/html"
 
# remote directory
RSYNC_DIR="/kunden/homepages/14/dxxxxxxxx/htdocs"
 
# excludes file - this contains a wildcard pattern per line of files to exclude
EXCLUDES="/root/rsync.excludes"
 
# the name of the backup machine
RSYNC_SERVER="leased-territories.com"
 
# username on the remote server
RSYNC_USER="uxxxxxxxx"
 
# command-line options
OPTS="-rlptDvz --delete -e ssh --exclude-from=${EXCLUDES}"
 
#####################################
 
# sync cgi tree
rsync $OPTS /var/www/cgi-bin ${RSYNC_USER}@${RSYNC_SERVER}:${RSYNC_DIR}/html/
 
# sync html tree
rsync $OPTS --exclude=**/cgi-bin/** /var/www/html ${RSYNC_USER}@${RSYNC_SERVER}:${RSYNC_DIR}/
 
# backup local databases
/root/backup_database.sh
 
# transfer databases
rsync $OPTS /root/phpbb_db.txt ${RSYNC_USER}@${RSYNC_SERVER}:${RSYNC_DIR}
rsync $OPTS /root/mt_db.txt ${RSYNC_USER}@${RSYNC_SERVER}:${RSYNC_DIR}
 
# remotely restore databases
ssh -l ${RSYNC_USER} ${RSYNC_SERVER} ${RSYNC_DIR}/restore_database.sh

rsync.excludes contains:

1
2
3
4
5
6
**/config.php
**/mt.cfg
**/mt-db-pass.cgi
html/jeff/gallery/.htaccess
**/usage
**/php.ini

Now, to move things in the opposite direction, I crafted a similar script, sync-from.sh:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#!/bin/bash
 
# local directory 
LDIR="/var/www/html"
 
# remote directory
RSYNC_DIR="/kunden/homepages/14/dxxxxxxxx/htdocs"
 
# excludes file - this contains a wildcard pattern per line of files to exclude
EXCLUDES="/root/rsync.excludes"
 
# the name of the backup machine
RSYNC_SERVER="leased-territories.com"
 
# username on the remote server
RSYNC_USER="uxxxxxxxx"
 
# command-line options
OPTS="-rlptDvz --delete -e ssh --exclude-from=${EXCLUDES}"
 
#####################################
 
# sync cgi tree
rsync $OPTS ${RSYNC_USER}@${RSYNC_SERVER}:${RSYNC_DIR}/html/cgi-bin /var/www/
 
# sync html tree
rsync $OPTS --exclude=**/cgi-bin/** ${RSYNC_USER}@${RSYNC_SERVER}:${RSYNC_DIR}/html /var/www/
 
# backup local databases
/root/backup_database.sh
 
# backup remote database
ssh -l ${RSYNC_USER} ${RSYNC_SERVER} ${RSYNC_DIR}/backup_database.sh
 
# transfer remote databases
rsync $OPTS ${RSYNC_USER}@${RSYNC_SERVER}:${RSYNC_DIR}/dbxxxxxxxx_phpbb.sql
 /root/dbxxxxxxxx_phpbb.sql
rsync $OPTS ${RSYNC_USER}@${RSYNC_SERVER}:${RSYNC_DIR}/dbxxxxxxxx_mt.sql
 /root/dbxxxxxxxx_mt.sql
 
# restore databases
/root/restore_database.sh

Originally I was planning on just changing (via mydomain.com) the DNS ‘A’ record for billimek.com to point to the IP address for my host on 1&1. Unfortunately it wasn’t that simple. 1&1 has some lame check to give an error if you try to hit your webspace from domain not in their system.

The only way to remedy this is to transfer the DNS registration with my registrar to 1and1.com. I did this Sunday afternoon.

By the time Sunday night rolled around, the change started to propagate through the root nameservers. I logged into my old Kennesaw shell and did a nslookup to find billimek.com pointing to 1&1’s nameservers. Lynx took me to the correct location so I sent myself a test email and got a bounceback with some strange error:

1
2
3
4
5
6
----- Transcript of session follows -----
... while talking to mx00.1and1.com.:
>>> DATA
< << 550 relaying to <jeff@billimek.com> prohibited by administrator
550 5.1.1 <jeff @billimek.com>... User unknown
< << 503 Valid RCPT TO <recipient> must precede DATA</jeff>

I figured that I needed to wait longer so I went to bed not too worried about this.

Nov 24th 2003

new tower

When I came into work today I did my daily check on the health of the six production tuxedo servers. At 10:11am the exchange valuation engine dropped a core on dpd22y, just as I was checking the ULOGs for errors.

Jegan did a quick dump of the stack trace and I started investigating. I loaded the core up in wdb and quickly located the ticket number. Back in production I ran the ticket-search script to look through all of the DTMO logs and found a candidate file. I ftped it over to devel and extracted the DTMO for the particular ticket. I ran the DTMO in sapitest and was able to successfully reproduce the problem.

I printed out a dump of the objects and determined the structure of the ticket:

NEW ticket containing two coupons