Jeff Billimek’s Journal

The past few days it’s been very crappy weather. This morning was no different. In fact it was raining so hard this morning that I hit a traffic jam on the south-bound lanes on the west-wall of I-285, something that never happens. A couple of times during my drive into work I hit a deep patch of water on the highway and started to hydroplane. That’s definitely something I don’t enjoy doing - especially when traveling at 75mph.

At 9:30am Jegan and I attended the “1.2″ managers meeting in 11East. Once again it was a learning experience. I was worried that the topic of our performance testing would come up. Fortunately when it was brought up, Alonzo glossed over it and didn’t press us for any numbers. Our performance test yesterday was inconclusive so we didn’t want to publish the performance numbers. In addition to that, the initial numbers we saw were so horrible that we didn’t want to publish them to a large audience until we can discuss them among ourselves.

After the meeting Jegan, Rajiv, and I had a frank discussion about the performance testing results we’ve seen so far. We shared with Rajiv some of the results and he agreed that the numbers appear to be pretty grim. We asked him what he thought about our proposal to combine all of the engines into one large controller. He also seemed receptive to that idea. He suggested that we call a meeting with him and Gus to go over the specific results.

For lunch today a large group of us went to Buckhead to a small place called the Taco Stand. We’ve been there once before and the tacos are really good. The two girls working the front counter were very cute too. The whole day it has been raining. When we were about to leave the restaurant, we ran from the front door to the car. When I started my sprint, I slipped (I was wearing dress shoes so there was no traction) in the water and went sprawling on my back. I managed to recover quite quickly and didn’t even spill any of my lemonade I was carrying. My back and pockets got quite wet though. My iPAQ was in one of my front pockets and I examined it for damage. Unfortunately it did not escape harm. The top portion of the PDA had a crack in it. It still works fine and the damage is purely cosmetic but I regret that it got damaged. I also have a quarter-sized scrape on my elbow.

After lunch a few of us attended a meeting called by Scott to discuss the 1.3 Official Proration engine. The requirements are pretty simple and it seemed straight-forward to me. During the meeting Casey suggested that we create a boolean indicator on the coupon proration entity to indicate if this prorate is official or not. The requirements call for us to duplicate the entire entity and just call it a new type. This made a lot of sense and we decided to go down this route. What bothers me is that I didn’t even consider that possibility. I take pride in being able to come up with ideas like that but I completely missed it here.

The drive home today was in heavy rain. I hit traffic resistance going along I-285 northbound on the west wall due to rubbernecking in two locations. When I was near I-20, Jenn called me and asked if I wanted to join her for dinner at Lenox. She gets off work at 6pm. I agreed and continued along 285 towards route 400. Fortunately I was going against the heavy flow of traffic so it wasn’t that bad.

Jenn and I had dinner in the mall at California Pizza Kitchen.

I brought my spare HD home with me to dump everything on it and plan on taking it back to work. This is part of my backup strategy. I do quarterly backups of everything to a spare HD and store it in my desk at the office. This satisfies my need for an ‘offsite backup solution’.

Because it’s a new month, I looked through my web server usage statistics. I made note of the top five google search hits for my site for the month of June:

[code]96 - Chicago Skyline Photos
49 - Sojourn
36 - the server was unable to allocate from the system nonpaged pool
23 - Chattahoochee River
20 - Amicalola River[/code]

It was a short day at work today. Because I worked so late yesterday I didn’t come in until 9am. As soon as I sat down, Casey told me that Carole was looking for me. Indeed I had an email from Carole (sent at 7:45am) requesting that Ram or I activate the build. Ram usually doesn’t get in until around 9:30.

I immediately began the build activation process but got stopped with an error from our currency conversion service. I went into the test lab and asked Lee to allow me to diagnose the problem. Logged in as the tuxedo domain user I was able to determine that an EBS service was not running, causing the problem with currency conversion. Last night the test server had a panic and rebooted. This is probably why not everything was running. I corrected the problem and got the system ready.

Unfortunately for the testing team, Chris Alain, the unix administrator for this server sent out an email warning us that the server was going down for emergency maintenance in about 15 minutes. I forwarded the message to Carole and Lee from the testing team.

For lunch today a large group of us went to Smokey Bones to celebrate Alex and Veena’s birthdays. Today I had a full rack of baby-back ribs. It was very good and I felt quite stuffed afterwards. Unfortunately my lunch cost me over $20.

The weather today has been gorgeous. On the drive to/from lunch I sat in the back of Susan’s large Yukon. On the drive I gazed out of the window looking at the beautiful blue sky and puffy white clouds.

After lunch Larry called me and said that going to San Diego this weekend didn’t look like a good idea after all. Amy had some specific plans. I told him that this was fine but he said he would talk to her and call me back if their plans changed.

On my way home from work I called Craig. I STILL haven’t purchased anything for his birthday which was in May. I called to see if he bought the NeverWinter Nights expansion or the audio CD for Harry Potter book 5. Unfortunately he already has both. He’s still looking for the name of a day-spa so I can get him a gift-certificate for a massage/manicure/pedicure type treatment.

When I got home I put on my swimsuit and went out by the pool. It was nice and sunny still this afternoon and I read my book and baked in the sun. Larry called and told me that he spoke with Amy and if I want, they would love to have me out in San Diego this weekend. I confirmed that I would be out there.

This evening Jenn called and invited me to spend the evening with her where she’s house-sitting. I gathered up some stuff and headed out towards the Lenox area. On my way to the I-285 entrance I passed over the Chattahoochee River and saw the river shrouded in a misty fog. Fortunately I had my camera with me (I try to take it with me everywhere in my backpack). I pulled over to the side of the road, grabbed my tripod out of the back of the explorer and walked to the bride over the river. I mounted the camera and took several exposures at varying apertures. I took a few wide-open and one stopped-down to f8 and another stopped down to f11.

I liked this one the best:

(15s, f11)

It’s the fourth of July today! Today we celebrate Independence Day.

I spent most of the day researching places to go for a day-trip tomorrow. Specifically, I looked at Chicago, New York, Boston, and Asheville, N.C. (where the Biltmore is).

The weather for Chicago looks quite pleasant. It’s going to be partly cloudy and in the 80’s. Unfortunately Delta reduced the returning flights tomorrow and so the last flight back is around 3 or 4pm. This doesn’t give much time to spend in the city. The train ride from the airport to downtown is like 45-minutes each way.

The flights to/from New York looked fine but the weather forecast calls for a hot-and-humid (in the 90s) mostly-cloudy day with a 50% chance of rain.

The flights to/from Boston look decent but the weather forecast calls for a hot-and-humid (in the 90s) cloudy day with a 40% chance of rain.

Asheville, NC is the home of the Biltmore estate. The forecast for the Asheville area is pretty nice. To drive there from Atlanta would take about 3.5 hours. Delta has a few flights to/from Asheville as well and that would also be a solution.

I ruled Chicago out due to the late return flight. New York and Boston didn’t seem to pleasant due to the hot, humid weather and chance of rain. I don’t think that’s conducive to walking around all day.

Jenn didn’t seem too keen on the Biltmore idea either. I also briefly looked at a solution to travel to the Outer Banks or the Savannah area but they weren’t possible for Saturday either.

Around 5pm I called Mom but she wasn’t there. I called Mom’s Mom, but she wasn’t there either. I also called Dad and spoke with him for a while. I told him of my plans to be in San Antonio next weekend and we made tentative plans to meet up.

I took a shower, packed my backpack with my camera and tripod, and drove out to the North Springs Marta station. I’m sure there is something closer but that’s one I know how to get to. I parked, bought two tokens, and boarded the southbound train.

While waiting for the train to leave, and during the trip down to Buckhead, I read my book. When I arrived, I called Jenn and began walking over to Lenox Mall.

On my way towards Lenox, I spotted a girl with a Canon EOS-1D (with the huge battery pack). I walked up to her and struck up a conversation about her camera. It turns out that she’s a photographer for the Atlanta newspaper. I was sort of surprised that they don’t use the EOS-1Ds. She said that all of the photographers at the AJC like the Canon’s and even a guy who used to be hard-code Nikon switched over. She had a 16-35mm lens and I asked her how she likes it. She said the extra wide angle really helps due to the crop factor. It was pretty cool talking with a real photographer.

It was about 7pm by this time and there were a lot of people already camped out in the parking lot. Most of the people brought chairs with them. We should have done that too.

I met up with Jenn after she got off work and we staked out a spot in the parking lot. She brought a large beach towel that we placed on the hot asphalt parking lot and sat down. Jenn went off to get some cash out of an ATM and some drinks. She called me a while later and asked if I wanted a hot dog and fries too.

When she came back we sat back and ate our food. It was pretty nice outside and some dark clouds formed on the horizon. Fortunately it did not rain but the dark clouds did bring a cool breeze.

Sitting next to us was a lady by herself. She engaged us in conversation and told us all about her work. She’s a psychologist and told us about some of her multiple-personality disorder patients.

Around 8:30 three Blackhawk helicopters flew overhead in formation and following them was a Delta 767 jet. Originally there were supposed to be two F-16 fighter jets escorting the Delta jet but they weren’t present. I presume the sight of a commercial airliner being escorted by two fighter jets would cause alarm to people not knowing that it was just for the festivities.

Finally around 10pm the lights in the parking lot went out and the fireworks started. I set my camera up on my tripod (legs collapsed, so it was very close to the ground where I was). I had already locked in the focus manually to some distant clouds as a reference point for infinity focus. I put it on manual mode with an aperture of f11 and a shutter speed of 4 seconds. I turned the beep off and set mirror lockup on.

The fireworks display was really spectacular and lasted about thirty minutes. During the show, I clicked off about 60-80 shots, not paying attention to the camera much as I wanted to enjoy the show. I was sure that I may get a few decent photos.

After the display, I walked Jenn back to her car and then turned around and walked back to the Buckhead Marta station. The train ride back to my car was uneventful.

When I got home, I noticed that the power had gone out. Unfortunately my internet connectivity was out too. I called the tech support line for my ISP but got an answering service which told me that they would get back to me on _Monday_. This is unacceptable to me.

I went through the photos and uploaded to them to an album on my photo gallery.

For most of the morning today I worked on the Exchange Valuation change request. There is something really soothing about coding (at least coding something you understand) and listening to music. It is quite therapeutic and rewarding.

Shortly before lunch, Jegan came by and asked me to work with him on putting the last touches on our performance modeling document. I had him send me all of the excel spreadsheets that I worked on yesterday and I embedded them inside the word document. I also pasted ‘links’ to the key charts for display as well.

Around 11:15 a few people started to gather around in the hallway in front of my cube. Brian and Casey were milling around asking about lunch today. I got up from my desk and joined them. They called Susan (who sits down on the 9th floor) and asked her to meet us downstairs in the lobby. James indicated that he needed to get cash and headed to the 4th floor to use the ATM machine. We walked by Alex’s desk and collected him. When we arrived at the elevator bank, James was still waiting for the elevator. I noticed that Brian disappeared. He went to the restroom. We had a couple of stops on our way down the elevator to the first floor.

When we reached the first floor, Susan showed up and then Brian and then James. As always we stood around asking each other, “What should we do for lunch today?”. Someone mentioned Schlotzsky’s but Casey didn’t seem happy about that idea. Casey then said, ‘Burger King or Wendy’s?’. I concurred with Burger King. We began walking out to the parking lot.

When we reached the parking lot I announced that I would drive some people. Casey volunteered to drive too. We drove over to Cleveland avenue to the Burger King next to the “Scary Kroger“. We arrived shortly before Casey. When we entered the restaurant, we studied the menu on the wall. I ordered an ‘All American Burger’ value menu. I only had a $20 bill in my wallet and the cashier didn’t have enough change. I had to wait a few moment while she got change. After getting my change I collected my empty cup and picked up my order which was waiting for me. I walked over to the fountain drinks and served myself some lemonade. I sat down at a round table with six chairs.

The others joined me and we ate our lunches and talked. I finished mine first and got a refill of lemonade. When everyone finished lunch I threw away my trash and got more lemonade. Casey had to run an errand at Home Depot so Brian rode back with us. On the ride back I had a Pete Yorn CD in my player and James recognized the song.

After lunch Jegan and I had a meeting with middleware regarding performance testing. It was pretty productive and the seemed concerned at the numbers we arrived at. Van suggested that we may be taking a performance hit due to Tuxedo’s compression kicking in when message sizes go over 10k. We agreed to keep them in the loop for future meetings.

When the meeting concluded I worked with Anil to incorporate Exchange Valuation into the derived data ’super controller’. We had some difficulties with the way Exchange was architected so I went over to Jegan’s cube to get his opinion.

On my way to Jegan’s cube I was ambushed by Chris from mid-tier and Dave from middleware. They asked me if I had time to talk. I said, ‘actually no I don’t’. Unfortunately this didn’t matter and they proceeded to have an impromptu meeting with me regarding our performance testing methodology. About ten minutes into the discussion Anil came by (looking for me) and got sucked in too. I got ‘emotional’ in some of my responses to their questions about the validity of measuring cpu seconds per process. It wasn’t that I didn’t think they had valid points, I was annoyed that I was the victim of a meeting ambush. After the ‘meeting’ concluded Anil commented that I might have been too passionate in my responses to them.

After this I sat down with Jegan and Anil to discuss how we can fit the Exchange engine into the model for the base controller. We looked at various options and I finally came up with an idea that they all seemed to like. I told Anil that I would implement the necessary changes on the performance branch so we can test it out.

Around 5:15 I changed into my softball clothes and drove out to the east point softball field. When I arrived, we practiced hitting and catching for about an hour until the game finally began at 6:30pm.

It was a close game. At one point we were up 8-3, but then the score got tied to 10-10. In the last inning the other team scored two runs, bringing the score to 10-12. When we were up to bat, we somehow managed to score 3 runs just before time ran out and barely won 13-12. It was a fun game (it’s always fun when you win). Brian wrote a short commentary about the last inning:

Great come back last night! Down 2 with 2 outs and nobody on…Chambers singles, but loafs to first base. Bob singles and Chambers still loafs to second. James singles, and now the bases are loaded with Hollywood Huggahalli up. Hollywood singles down the left field line and two runs score and then James somehow beats out a pickle to score and we win 13 -12 to keep our playoff hopes alive.

On my way home I stopped at Publix and got groceries.

For most of the day it was a pretty calm and normal Thursday. After lunch at 1pm we all headed across the street to the ‘training center’ building where they train flight attendants. Inside this building are mockups of the insides of planes and a giant pool where they practice water landings. Also inside this building is a large auditorium.

This is where we had our ‘Revenue Pipeline’ all-hands meeting. It lasted about two hours and the first part was (to me) fairly uninteresting. The last part of the meeting was a trivia session conducted by Brent, our VP. We all broke into teams of six people and all answered ten questions. There were about nine different groups. The group I was in ties for first place and we had a runoff question but lost.

At 4:30 it came time to go to the mysterious meeting I found out about yesterday afternoon. I headed over to 11East along with the other people. James was absent and Brent had him paged. It turns out that we have exactly fourteen days to come up with a GUI displaying real-time stats about our project, Revenue Pipeline. Because it is a backend accounting system there isn’t a lot to ‘display’ about it. This team of six people is supposed to come up with something that looks slick so the executives can demo it to an outside organization Thursday after next.

That’s not a lot of time and the ‘requirements’ are fairly nebulous. Brent assigned us different type of tasks to work on. Curtis and I are working on the GUI aspect of the project. James and Elizabeth are working on the Data portion. Ningyu (an advisory developer for the scheduling project) is going to assist us with a product called BusinessWorks. We’re going to meet with him tomorrow morning. Jeff C. is going to be doing the project plan.

I was a little concerned about the group selected and spoke with Brent about it in private after the meeting. He put my concerns to rest. I’m going to have to step up and work really hard to do well on this project. It’s going to be pretty busy for me the next two weeks.

I called John and coordinated with him about this weekend. He’s going to be in Austin having lunch with his dad. I’m going to try to join them if I can.

I got a pretty early start today. I went to work around 7am and spent the morning wrapping up the coding and testing for the partial reissues change request. I checked in something like ten files into ClearCase related to this change.

I then made several phone calls to people in other groups so I could inquire about the details of some of the metrics and instrumentation log files used for our project. I’ve been trying to get a handle on what exactly we have available and how often it is updated.

Curtis and I met with Ningyu at his desk on the 10th floor. He spent about an hour and half with us demonstrating what the Tibco BusinessWorks product can do. It’s pretty slick and I think I should be able to pick it up fairly quickly. He directed us to the ClearCase vob location where we can find the installer to install the designer on our own workstations.

Because the meeting with Ningyu lasted so long I missed lunch. In drove down the street to McDonalds and got two big macs to take back to my desk. I ate at my desk and went through some of the BusinessWorks documentation.

Today Jegan is running a performance test. I popped my head into the test lab a few times throughout the day to see how things were going. He seemed a little stressed but the test appears to have been successful with no problems. Chris and Dave were on a conference call with Jegan and we discussed postponing our scheduled 2pm meeting since they were actively monitoring the test.

At 3pm I went downstairs to the 10East conference room to attend my first ever ‘PAT’ (Portfolio Architecture Board) meeting. Not many people showed up and I had to leave at 3:30.

Jeff C. set up daily meetings to discuss the special project for Brent. Today we met in 11West at 3:30. The meeting was scheduled for an hour but we didn’t leave until around 6pm. During the meeting (Brent was not present) we wrestled with the idea of presenting a real-time solution. Several of our metrics and statistics are only updated every thirty minutes. We then looked into the idea of having a ’simulated real-time’ presentation in which we display the statistics for a 24-hour period in the span of five or ten minutes. Alonzo (my director) said that he would check with Brent on this question and several other questions we came up with.

My iPAQ PDA woke me up around 8am and I snoozed until 8:30. I woke up, got a shower, dressed, packed, and checked out of the hotel. Considering I was there in the hotel room for less than seven hours, it didn’t feel very satisfying that it cost me $90.

I drove over to Grandma’s house and Barbie was already there. We ate breakfast (which grandma cooked). She served fried eggs, bacon, and buttered toast. It was delicious!

After breakfast we sat and talked for a while. It was nice seeing Barbie again. Around 10:15am Barbie and I went to the care facility where Grandpa is staying. We visited with him for about thirty minutes. He’s been bedridden for the past eight years ever since his stroke. He’ll have his 87th birthday this next Sunday. He seemed to be doing pretty good given his condition. It was nice to see him again.

Barbie went home and I stopped by Grandma’s house again to talk with her for a little while. Finally around 11am I left to drive to Austin.

On my way to Austin I was traveling in the left lane going about 80Mph (the speed limit was 70). I was talking on the phone with Jenn and noticed that I just passed by a state trooper pulled over on the side of the road. In my rear-view mirror I saw him pull out onto the highway. I got off the phone with Jenn and changed lanes into the right lane (and slowed down), hoping that he might not notice me. I was pretty certain he was coming after me since I was the only one in the left-lane when I passed him.

About a minute later I saw him coming up fast on the left. I just knew he was going to pull behind me and flash his lights. Fortunately that didn’t happen and he pulled behind a truck (the same color as my rental car) two cars in front of me. He pulled over the truck. What a relief I didn’t get pulled over!

I drove to Dad’s house and we sat outside in the backyard talking for about an hour and half. Around 2:30 we went out to a Mexican restaurant for a late lunch/early dinner. It was nice seeing Dad again and he seems to be doing pretty well. Tex (the big black Labrador) was rambunctious as usual and Hicks (He’s about 14 years old now, we got him when he was a kitten) was just as wonderful as I remember.

I left Dad’s house around 4pm and drove to the airport. I dropped the car off with no problem and got assigned seat 14E in coach class. The flight back took longer than normal because we had to fly around a large storm. Once in Atlanta it took me like 25 minutes to get to my car because the shuttle busses were slow.

I didn’t get home until about 9:30pm. I uploaded photos from this weekend. I wasn’t too impressed with some of the shots I took around UT. I stacked the polarizer on top of the UV filter instead of switching them out and I think this distorted some of the image quality. I uploaded a few photos into a special photo album for UT.

I skipped my 9:30am managers meeting this morning so I could spend more time working on the special ‘quickdraw’ project.

At 1:30pm we had a meeting in the Jacksonville conference room. We discussed, in detail, what sort of business metrics we want to display. The human factors people were here again and gave a lot of input.

In this meeting we decided to use perl to parse the two data file and update the database directly instead of going through BusinessWorks. This is fine with me, but I was quite annoyed that I spent a lot of time working on the BusinessWorks solution and now it’s not going to be used.

Immediately after this meeting I went to a performance testing status meeting with Carole. Representatives from mid-tier engineering were there. We went over the numbers gathered from the performance test last Friday. We were able to come to the conclusion that we can do about 1.6 transactions/second per 8-cpu, 16-gig ram HP n-class server. In the production environment we’re aiming for a throughput of 19 transactions/second. Obviously we’ll need a lot of servers to do this. Everyone in the meting recognized that this is a real problem and we need to look into additional hardware.

After this meeting I dashed to the Jacksonville meeting room on the 5th floor for a 5pm meeting. Jeff Chambers was there with his two kids because he’s going to be leaving early for a softball game. Brent wasn’t at this meeting.

We talked mostly about the details of the GUI and what we can and cannot do. The human factors team came prepared with a printout of a prototype. Because BusinessWorks is out, I’ll be working with James and Sameer on the web server side and helping create the JSP code to grab the data.

When I got back to my desk at 6:30 Jenn left me a voice-mail asking if I wanted to go see a movie with her. I left work and drove up to Lenox. We drove down the street to Phipps plaza and had a snack in the food court before heading to the theater.

We got tickets to ‘Charlie’s Angles: Full Throttle‘ and watched the 7:40pm showing. It was a pretty fun movie. All of the stunts were completely over-the-top and not realistic at all but it was still good.

I got an email from Michele inviting me and my girlfriend over to her new house to have dinner with her and her boyfriend. I floated this idea to Jenn and she agreed to go.

I made it into work around 8:30am today. This morning I worked with Sameer and James on the Java servlet coding for the server-side potion of the GUI presentation layer.

We had an 11am status meeting but Brent was unable to attend. The meeting was fairly short as we were all working hard on the coding piece. Alonzo encouraged us to continue working hard. Everything is pretty much coming together now. The perl code to read the files and populate the database tables is nearly done. The middle server piece to query the database and assemble the results is also coming along quite well. And the GUI presentation piece that the human factors team is working on is also pretty much complete too.

I had an ad-hoc meeting with Jegan, Carole, Rick, Gus, and Anil regarding performance testing.

I spent the rest of the day working with Sameer on the Java servlet code. Sameer was doing all of the coding so I just watched and asked questions.

We had our 4:30pm meeting in which to present everything working together. Unfortunately there were some javascript errors giving us problems which prevented us in presenting the final product at the 4:30pm meeting. Brent wasn’t present at that meeting either. Alonzo asked if any of us would be willing to work over the weekend to get it finished. Elizabeth, James, and Sameer volunteered to come in the office tomorrow. I volunteered to help all-day on Sunday. Alonzo stated that Brent will be out of the office on Monday so we will need to give the final working product first thing Tuesday morning to make sure he is happy with it.

I didn’t leave work until 5:30pm. It was only an 8-hour day today, I felt guilty for leaving so early - but I had plans. Jenn didn’t want to drive down to my area to start our trip. This would have saved me an hour driving time. I drove thirty-minutes north to pick her up and then turned around and drove thirty-minutes back south again - to the Airport.

We took I-75 south to I-16 to I-95 and then to a smaller highway to reach Hilton Head. On the way we stopped at a Wendy’s in Morrow for dinner. It was a long drive (took about 5.5 hours) and there were parts that went slow. With the afternoon thunderstorms, it was raining so hard I could barely see the road. Jenn and I joked about all the other drivers turning on their emergency flashers while it was raining. I wonder if it’s a requirement to turn on your flashers whenever it rains.

We finally reached our destination: The Holiday Inn Express. It was only $69/night (before tax) probably because the hotel wasn’t on Hilton Head island. The other place I looked at originally, the Crowne Plaza Resort was directly on the beach but the rate was $240/night.

It took a long time to check in. I stood in line for about twenty minutes while the girl at the front-desk checked people in. This Holiday Inn Express certainly didn’t feel very ‘express’ tonight.

I got into work around 8am. We had a 10am meeting to discuss the status of our special quickdraw project. Over the weekend we found out that the queries against the two main tables were taking too long. At a suggestion from our DBA (Andre), we created summary tables in order to consolidate all of the old data.

For lunch today just Alex and I went to Wendy’s. Casey was in a meeting across the street until around noon and Brian and James were in an RDMF lunch-meeting. Bob wasn’t around. During lunch Alex told me about his air-conditioner problems with his home. The problem is with some coil and it’ll cost him $700 to fix. He’s excited because he and his family are going to Tybee Island for a few days.

When I wasn’t in meetings today I worked with Sameer on writing the java servlets. We were really disappointed with our approach to represent total coupons sold as a scatter plot graph. Sameer and I toyed with using the VML ‘polyline’ function to draw a ‘line graph’ instead. From our initial tests it was MUCH faster than the scatter plot. The scatter plot was so slow that it took around five minutes to paint in the browser. This simply wasn’t acceptable.

We had another quickdraw status meeting at 4:30pm. In the meeting Sameer and I presented our idea of the ployline solution. Everyone liked the idea. We worked the rest of the afternoon to implement it into the GUI. I left work around 7:30pm.

Jenn met me at home with some groceries. We both cooked diner together. We had bow-tie pasta with parma rosa sauce, a ‘real’ salad (that Jenn prepared), as well as buttermilk biscuits. It was a really pleasant dinner.

I got into work at 7:30am. As soon as I made it up to the 11th floor I started pouring rain outside. It rained very hard for about fifteen minutes and then suddenly it was sunny & blue skies. It was cool how such an isolated storm blew through so fast.

The ‘live’ data feed was turned on this morning. In other words this was the first time we saw the twenty perl scripts parsing all twenty files in realtime. The result of this was twenty different database updates taking place across two tables. Fortunately the frequency of the updates wasn’t saturating the database but it certainly had an effect.

Even though we have summary tables and have summarized all of the previous day’s worth of data, the two main tables are very large (about 11 million rows). When we switched the servlets to query against the real tables (now that we have a live feed) we saw very horrible performance. Just as we noticed this, we had to attend a 10am meeting.

During the meeting we explained the dismal query performance. We came up with a few ideas of things to try and planned on deleting all of the rows for the data currently summarized. Jeff Chambers had set up a computer with the 40-inch LCD display panel to demo the GUI. We dismissed the meeting early to continue work on the SQL.

I skipped lunch so we could continue working on the problem. I got a page from Alonzo asking for me to come to the 11East conference room. When I went inside there were a lot of important people there. The first one I noticed was the CEO of Delta Technology and CIO of Delta Airlines, Curtis Robb. Along with him were my senior vice president Mike Childress and another senior vice president. My vice-president Brent was driving the GUI. Fortunately it was working (but not updating due to the queries taking so long). I walked up to Alonzo and he mentioned to me that they want to get the computer out of kiosk mode in order to pull up the Brio reporting tool.

Curtis and the senior vice-presidents were very impressed with the GUI and the real-time analysis that it represented. Brent seemed to be having a good time explaining everything as well. When Brent asked how to get the computer out of kiosk mode I pulled up another instance of internet explorer for him. While he was demonstrating the reports and more importantly the GUI, I had a great sense of pride. All of our executives were very impressed with the work that we did over the past week.

When they dispersed I went back to working with Sameer. While we were waiting for the old data to be truncated from the database, I took the opportunity to set up an instance of Tomcat on our HPUX development unix server. I worked with Andy Mitchell from release engineering to get the installed version of tomcat imported into our clearcase vob as a set of clearcase elements. Sameer sent me a copy of all the files required to run on the webserver and I configured tomcat. Once I got this working and tested everything, Sameer and I started testing on this server instead of his workstation.

We noticed the queries still taking a long time. This didn’t make much sense since the HPUX server is an 8-processor machine with 4-8GB of RAM. Unfortunately it seems that we have to much activity on the server there aren’t enough resources to devote to java as we would like. We then decided to move back to the tomcat instance on Sameer’s NT4 desktop workstation.

It was frustrating to find out that the now-truncated table was still taking about 5-6 minutes per query. This was still entirely too slow. We made some more tweaks to the servlets and rebooted the tomcat server. I walked over to the 11East conference room to restart the client for the GUI. I knew that the slow response times would mean that it could take thirty minutes to come online.

When I was about to leave the conference room Brent walked in and informed me that Curtis and Mike were coming back up to take another look at the GUI. When he told me this, my heart sank because I knew it would be quite some time before the GUI was running. I asked Brent to stall them as much as possible to buy us time.

Unfortunately Curtis and Mike walked in shortly after that. They questioned why the GUI wasn’t running and I explained that we’re tweaking the database queries still and it takes some time to load the very first time. They decided that they would just wait. The minutes ticked by as everyone stood in front of the display’s white screen waiting for something to draw. Fortunately Brent offered to demo the Brio reports while we wait.

When Brent did this I went over to Sameer’s cube to update him on the situation. He seemed pessimistic that the GUI would come up at all. I went back into the conference room dreading what would happen if they couldn’t see the GUI again. After a few more minutes Curtis asked Brent to switch back to the GUI to see if it loaded yet.

I held my breath as he switched to the browser. I through for sure it would still be a white screen and there would be questions about why it isn’t working now. Fortunately by some stroke of luck it was online and they could see everything.

With a potential disaster averted, I took a short break and saw Jegan in the hallway. We discussed a meeting scheduled for tomorrow to talk about hardware planning for the 1.2 release. Carole has been out of the office and doesn’t know about the meeting and won’t be in tomorrow. We were concerned about some of the people invited to the meeting and the perceptions some of those people might have about how the applications team is handling things. We decided to call her.

Jegan and I got on a conference call and called Carole at home. We updated her on the situation and she thanked us for the call. She suggested that we talk to Alonzo before the meeting to let him know our point of view on the situation since he’s running the meeting.

When we got off the phone with Carole I called Alonzo and proposed meeting with him before the meeting. He mentioned that he was going to reschedule the meeting for Monday so that Carole will be able to attend too.

By this time it was around 6:30pm. Sameer and I were waiting for the DBA, Andre to run an errand and come back to the office to analyze our queries to see why they are taking so long.

While we waited we brainstormed about possible solutions to the problem. Jegan came by at one point and asked about us running the queries in parallel. I disputed the idea thinking that three selects against the same tables at the same time would certainly take more time than running them sequentially. We decided to do an experiment. To my great surprise it took about seven minutes to run the queries sequentially but only 3.5 minutes to run in parallel! This looked like the break we were looking for.

Sameer and I spent much of the rest of the evening trying to get the servlets to run in parallel. No matter what we did, they would always execute in sequence. At one point we even had the crazy idea of hosting two of the servlets on my workstation to do a form of distributed computing in order to get them to run in parallel. This did not work either.

We finally found a way to make them run in parallel by re-architecting the HTML and javascript such that the servlets are all in their own hidden frame instead of one frame. Ours HTML knowledge wasn’t so great in this area so Sameer called one of his friends for some tips on the syntax.

It was now around 10pm and Andre finally called us back. He did some analysis and told us that he could ‘borrow’ 300MB of memory from another database to give to ours since our queries are so memory-intensive. Sameer also requested the primary keys be given another index to improve the query time as well. Both of these changes would require the database to be bounced. We didn’t want to disrupt the data feed still running so we decided to do this first thing tomorrow morning.

We completed our work on the server side and finally left work at 11:30pm. I had been at work 16 hours and was pretty tired.

When I got home, I cooked up a Totinos pepperoni pizza and fired up the VPN to get into work. Elizabeth asked me to check in on the perl parsers after midnight to make sure that they didn’t stop working when the old files got archived at 12:10am. I checked in on this and everything seemed fine. I went to bed at 1am.