Insomnia

Sometimes, ignoring your gut might be the thing that you do that requires the most experience and intuition. I find that this is a rule that often applies when trying to solve a software performance problem.

You see, my current project is a business intelligence and reporting product powered by Microsoft Analysis Services. I have spent a while on this project an I am getting pretty good at understanding what happens under the hood of the engine when the application performs queries. I have also spent a lot of time so far tuning the data cube to allow the application to achieve a good level of performance.

Recently, my customer started using the application with real-world data and was puzzled by the response times he was getting from one of the datasets. It was a rather small dataset with nothing out of the ordinary. One view, no calculations. Something really basic. But every time you performed an operation, you had to wait for an unusual 5-7 second delay before getting responses from the database.

I spent a day working with that dataset, massaging it, creating indexes and aggregations. pushing and prodding every which a way. Applying the same recipes that got me some level of success in the past. But nothing happened. I knew that what I was doing was improving things because I could measure the imrovement on the other datasets but this one remained very sluggish.

So I resigned myself to write to my customer with a hypothesis that I had about the reason why this dataset was slow (lots of "zero" values instead of nulls... that was my hypothesis... really lame in retrospect). As I was writing, I spent the time to explain the things that I had tried and, since my customer is smart but he is not necessarily an MDX query expert, I "took id down a notch" on the technical side. Taking care of going through all the details and all the steps. And as I was explaining, It struck me that I didn't really do all that I was writing. I didn't examine the data to see if it was different, I just quickly glanced and the text files containing the raw data and gauged the complexity of the dataset by their number and their sizes.

So I took a step back and I went back to the basics. I carefully examined all the dimensions of my data cube to realize that I had one dimension with a really large number of members. A 100:1 ratio compared to the other datasets that I had imported in the application before. It was so simple. The total "size" of the imported data files were similar but the "shape" and size of the resulting data cube was radically different. The way the queries were constructed made assumptions that were wrong. After that was identified, it was easy to come-up with a plan and test a few queries to validate my findings. All of this would have taken an hour if I had spend the time to go through all the steps.

Today, my experience and my knowledge was my worst enemy and it led me on a wild goose chase. When I started questioning my assumptions, the answer was staring at me right in the face. And yes... sometimes, you have to check if the computer is still plugged in.

Labels: work

• 0 Comments

Recently Matt was telling me that there was a certain amount of backlash in "The Valley" against working with Indian or Chinese colleagues. Not because they are not competent. But because it required people in California to stay at the office late in the day to be able to attend meetings with their Asian colleagues.

Interestingly, on the east coast, the situation is a little different. We benefit from the willingness of our colleagues in India and Eastern Europe to stay late for meetings (sometimes quite late). In order to be considerate to our colleagues overseas, we scheduled our daily SCRUM meeting early in the morning. By early, I mean 9:30.

Now, what is starting to happen is that our employees are complaining that this meeting does not match the core hours of our flexible time policy. The core hours are described in the employee manual as 10:30 - 15:30.

So begins the clash between HR needs and business needs. We need the help of our overseas colleagues to complete projects because there is not enough local talent to answer the demand. And we need to provide a comfortable, inviting work environment for our employees in all of our offices.

Anyone has smart ideas on how to solve this?

Labels: work

• 3 Comments

Those that have been reading my blog for a little while will remember that my wife has been having all sorts of trouble with her HP desktop computer with Vista on it. The reliability of the machine has been so bad that I decided to install the release candidate of Vista SP1 when it became available. That improved things slightly but the computer still couldn't properly shut down 2 times out of 3.

Well, the release candidate started complaining that it was not going to be supported anymore and that I had to install the real thing. So, on a rainy morning this weekend I started the process that would eventually take over 4 hours of my life.

The first step was to remove the outdated version of the the service pack. After some Googling to make sure that I was removing the right update I started the removal process. The first time, the machine froze and it had to retrieve a restore point to recover. The second time, the removal succeeded. But alas, the service pack did not show up in the Windows update dialog. Instead, Vista informed me that there were 19 updates to download and install first. I started the process and the computer froze at the 3rd update. I had to force a reboot of the machine and Vista retrieved another restore point. Back to square one.

Looking at the install log yielded no useful information. I turned to the HP web site for help. After describing the computer model and navigating to the support and downloads page. I was informed that I had to update drivers for SP1 to install properly on this hardware. No less than 6 driver updates were critical or recommended for this particular model of computer. From the Video card to the SATA drivers without forgetting a good old BIOS update. Each one of those updates had to be downloaded and installed individually. And each one required a reboot after installation.

An hour later, I was able to apply the 19 updates that would not apply before. After those 19, there were 3 more that appeared. And finally, Windows update offered SP1 for my installing pleasure.

So why the big rant? Is it just me or was this a really big missed opportunity by the hardware vendor? That PC came with a disk full of crappy software I didn't want. What if HP had invested some money and offered real value to their customers? What if they invested to streamline the delivery of critical driver updates. The same way Microsoft does it with its Windows updates. Something that a normal consumer, one without a degree in computer science, could understand. Something that didn't require me spending 4 hours on a weekend watching my computer reboot.

Interestingly, my Dell laptop had a different issue with a similar resolution. I was having trouble with the sleep/hibernate functions and I had to update the BIOS and Video drivers from the Dell site. The process was not much easier but at least, the hardware didn't hang while applying updates.

Labels: Vista

• 0 Comments

I was reading this article about the shocking steps that General Motors was taking to bootstrap their electric car project (called Volt). To allow themselves to innovate, they brought back a senior engineer from Germany and they threw all their processes out the window. The rest of the article describes how they peeled-off the industry's assumptions about car making (in general) and electric cars (in particular).

To me, one of the most interesting things that this article is the fact that they eventually realized that they cannot afford to just keep up with the Japanese car companies. They cannot play it safe. They had to "leapfrog" the competition and force them to follow their lead. This is the only way they could regain the technological edge.

The Volt is not available to customers yet. And it might still be a while before it is; But, according to the article, other car makers are already using the Volt's "technology" to design a new generation of electric cars. Even if the Volt never becomes a commercial success, the strategy already paid off for GM as it re-established them as innovators in their industry.

Labels: design, work

• 0 Comments

Maybe this is old news to most people. But this is the first time I see this video. It is a little scary.



[Edit] Many people say it is a hoax. Here's the page from Snopes.

Labels: fun

• 0 Comments