Search The Web

Today's Headlines

Sunday, September 5, 2010

All Is Well That Ends Well!

This week, I encountered an unexpected problem at work. A data analysis program that I am partly in charge of supporting started producing meaningless results, and nobody could figure out what was going on. This was an old model that was written sometime in the latter half of the 90's (the comments in the program indicated that the program was started sometime in 1997, but there were comments from much later also).

The program had been validated and tested, and around 2001 or 2002, it was basically in pure support mode. The program consisted of a SAS part that produced a forecast of some sort and then a C program that used these forecasts to do some optimization. I am in charge of supporting the C program, and one of my colleagues is in charge of the SAS programs. Neither of us have delved deep into the program because we took over support of the programs long after the program was in pure support mode.

By that time, all the original authors of the program had left the company. We had had a little bit of "technology transfer" from the authors telling us the basics, and since the programs developed no problems most of the time, we never had reason to dig into the source code that much. Given that we were busy with other projects, etc., these programs were pretty much forgotten, with the assumption that they would continue working with no problems going forward.

Well, the program decided to break down this week. Initially, the results from the optimization program became nonsensical, so the users assumed something was wrong with the optimization part of the problem. We dug into it a little, and quickly found out that the reason the optimization program was producing nonsense was because it was being fed nonsense by the forecasting program. Garbage in, garbage out!

The forecasting program, as I mentioned earlier, was written in SAS. Neither I, nor my colleague who is actually "in charge" of the forecasting program is a SAS expert. She knows a little bit of SAS whereas I know pretty much nothing. I am good at SQL and other components used in SAS, but in general, I find the syntax of SAS quite off-putting. I am used to procedural languages like C, and the SAS way of doing things with tables was not very intuitive to me.

Moreover, the program was not very well written. Because of space constraints, memory constraints, or out of plain malice, this SAS program took spaghetti code to new heights. Tables were dropped and recreated at will instead of different datasets being in tables of different names. Variable names were reused all over the place, and dozens of external SAS macros were called from within the program. Tracing the working of this program was a bear.

The problem started out on Monday, and nothing we did solved the problem for the next few days. We were getting desperate, and our lack of deep SAS knowledge was hampering our debugging also. We were getting resigned to the seemingly inevitable fact that we would have to leave the problem unsolved over the long weekend, and come back and work on it the next week.

I hate doing that because I know that I will worry about the problem incessantly when I leave it open over a long weekend. Already, every day when we left the problem for the next day, I found it hard to let go of the problem, and on a couple of days, I had dreams about the program and its problems! I could only imagine what havoc an open problem like that would wreak on my peace of mind over the long weekend. The user had to produce results for our company's CFO from the program, and he said he needed the results yesterday, but no pressure!

Fortunately, at about 4 PM on Friday, as we toiled over another table printed out from the program in our attempt to trace the data backward all the way to its source, I subconsciously noticed that something was off with the numbers in one of the columns. At first, I couldn't make sense of what I was seeing. Nowadays, because of old age or whatever, thoughts take a lot longer to crystallize in my conscious brain after they register with the subconscious.

I was pacing back and forth outside the cubicle of my colleague who was following some other lead, when finally it hit me. We quickly checked in the source code and found the calculation that was causing the problem. It was quite subtle and it was easy to miss. In fact, the calculation in the code had worked for several years without problems, but it would fail when the data was just so. And in this case, the data was just so, triggering the bug, and causing the program to spew out nonsense.

We quickly put a fix in place to cause the program to treat the data correctly, and the results miraculously became well-behaved. We could not believe our good luck. The user had already left his office for the long weekend. But we wrote him an email explaining that the problem had been fixed in development and he could get his numbers whenever he came back to the office out of the development system. We would arrange to put the fix in production later next week.

We heaved a big sigh of relief, and left for the long weekend at about 6 PM on Friday. It took us a while to test the fix and make absolutely sure that we had fixed it for good. We certainly did not want to be confronted with a fix that needed a fix on Tuesday. Our manager was also very relieved, and we all left for the long weekend in much higher spirits than we would have if we had not had the good fortune of finding and fixing the bug. It took us most of a week, and we would have preferred it if we had found and fixed the bug on Monday, but better late than never, I guess. So, all is well that ends well as far as this rude intrusion into my work week was concerned.

The weekend has been quite relaxed and peaceful, partly because of this good outcome. Yesterday, we had a family outing to a nut store where we were able to sample all kinds of nuts, candy made out of nuts, etc. The stuff on sale there turned out to be not of very good quality (the good quality stuff was not on sale, and it was no cheaper than buying it in a regular grocery store), but at least it was something out of the ordinary for us.

Later yesterday evening, I took my kids on a walk through the neighborhood. I have been going on long walks in the evenings the past several days, and I wanted to take them on one such walk. It turned out to be not such a good idea. They hardly interacted with me, instead choosing to yak amongst themselves endlessly and chasing after butterflies and moths they encountered along the way. I had to keep shouting after them to keep them from running too far ahead or falling too far back. All in all, it was like taking a couple of cats on a walk without leashes on them!

It is not like that when I take just one of them on a walk because then they have nothing else to do except listen to my bad jokes and interact with me. So, I have decided that perhaps I should take them on walks individually, alternating between the two of them rather than taking both of them out together.

I published a couple of post on my blog this week. Both of them were about my Peru vacation, which is now almost a month old. Accounts of day 8 and day 9 both went out this week, concluding something that had just hung there unfinished for the longest time. Now, I just need to get back to my regular publishing about other subjects until I do something exciting, such as taking my next vacation.

There were visitors from two new countries to my blog this week. They were from Montenegro and Mali. With that, the number of countries from which I have had visitors stands at 148. The chart below shows the breakout of visitors from the top few countries in terms of visitors to my blog.

It is amazing to me that my blog gets more visitors from Bhutan than from 140 other countries (Bhutan contributes the 8th most visitors to my blog according to this chart below)! Bhutan has a population of less than 700,000!! It is ranked 163rd in the world in terms of population, yet it is ranked 8th in terms of visitors to my blog!!! Wonders never cease . . .

Breakdown of visitors by country
But it did pique my curiosity, so I started doing some research into Bhutan. I had only seen photographs in the past, and knew it was a beautiful, mountainous place. But I never cared to find out more about it, such as its history, culture, etc. It is a fascinating country with a very unique distinction: the highest point in Bhutan is Gangkhar Puensum, which has the distinction of being the highest unclimbed mountain in the world, at 7,570 metres (24,840 ft). I hadn't even realized that there are unclimbed mountains in the world!

I am no mountaineer, but given the successful assaults on much higher mountains, I started wondering what was so special about this mountain that it would remain unclimbed. It turns out that mountaineering in Bhutan has been prohibited since 2003. Between 1994 and 2003, climbing to peaks higher than 6000 meters (20,000 feet) was prohibited out of respect for local spiritual beliefs. Bhutan was open to mountaineering only in 1983, and only 4 attempts were made on this mountain between 1983 and 1994. The first expedition never even found the mountain because of bad maps!

Bhutan is also the first country to advance the concept of Gross National Happiness, a measure of the quality of life or social progress in more holistic and psychological terms than gross domestic product (GDP). Bhutan's central planning process revolves around the GNH measure rather than around the country's GDP and other measures of economic development. Maybe, one day, when I get out of the rat race I am trapped in, I will settle down in Bhutan or some place like that which actually values the happiness of its citizens!

If it were any other week, I would be scrambling around this time to get all set for another week of work. But this being the labor day weekend, I have one more day to spend at home before I have to head back to work. So, happy labor day to everyone out there, and to myself!

No comments:

Visitors Country Map

Free counters!

Content From

In the News

Article of the Day

This Day in History

Today's Birthday

Quote of the Day

Word of the Day

Match Up
Match each word in the left column with its synonym on the right. When finished, click Answer to see the results. Good luck!



Spelling Bee
difficulty level:
score: -
please wait...
spell the word:

Search The Web