To Find the Unfindable Bug

This morning I have been tasked with reproducing an unreproducible bug.

Since the last update of our software, we've heard from a very small number of new users who were unable o create an account. This is a big deal for this small percentage of people. It is possible that the number of people that are reporting this problem are a fraction of the people who are experiencing it, since many people will just drop-off if they can't create an account, but even so, we also know that most people don't see this problem. We've also not seen this problem during extensive testing.

Except that we have seen this problem in testing. A few times in dozens and dozens of attempts. Each time, the exact steps were recreated, and it worked fine the next time. Its what we in the testing business call "a fluke".

But, back to the business at hand - how can I solidly recreate this most elusive of bugs? The first thing to do is to find out all we can from the people that experience the bug. Using our best detective skills, we should try to determine how the people who see the bug are different from those who don't. It could be something about about how they are interacting with our product or something about their environment (physical resources, other software, even their physical location). If you can isolate what the difference is, it should be easy to reproduce.

If you can't isolate the difference, then it is time to bring in your testing skills. Two approaches you can try:

1. "The Crazy Uncle" - Test unusual user flows. Since most people don't see the bug, perhaps we need to try something weird and unexpected to make it occur. Never underestimate a user's ability to use your software in a way that is totally unimaginable.
2. "The Big Snafu" - Make what can go wrong, go wrong. This means limiting or disabling your systems resources while going through the use case. This could include such things as unplugging from the network half way through, removing the install CD at the wrong time, or devoting but a sliver of your system's RAM to some application other than what you are testing.
3. "Brute Force" - Test the usual flow many many many times. It could be what is happening is some sort of race condition that will only occur 1 out of 100 times. This approach is extremely time consuming and boring (and therefore a good candidate for automation), but if you can make it fail when everything else is normal, that is important information that the developer can use to fix it.

If after all this you still can't reproduce the bug, then your choices are to keep trying, live with it, or hope the developers can find it by scrutinizing and carefully reviewing the code. None of this is especially appealing, but no one promised you a pleasure cruise.

Does anyone else have suggestions on ways to find the unfindable bug? Please add them to the comments if you do.

One of Those Days

Some days you get in a zone where the bugs come out of hiding to follow you like you were the Pied Piper of Defects. For me, that describes exactly how this day wasn't.

One thing I was testing involved passing strings of text. Of course all the basic, simple text worked fine. I also tested such odd ball characters like brackets and quotes and other punctuation. I even tested non-English characters including € and the like, and even some Japanese characters. Everything I tried worked perfectly, so I declared it good to go.

My boss then decided to give it a try. He happened to choose some text including a $. Of course, it failed miserably.

Later I was testing a new widget. I tested in on FF on Windows, and found some issues. These were fixed. I tested it on IE7, found some issues, and these were fixed. I tested it on FF3 Beta on Mac, and it worked there. It was good to go. Or would have been, had it not been on Safari. Guess who tried on Safari? That's right - my boss.

Sigh.

Some days you get the bugs. Some days the bugs get you.

Pay Back Bad Developers

Here we are, in the thick of the holiday season when we give gifts to the important people in our lives. When doing your Christmas shopping, you should be careful not to overlook the founders of the feast - the Bad Developers.

I say this not because developers write the code that gets sold to your users, thus paying the bills. I say this because without Bad Developers, and the wonderful bugs they tuck away in the product like lumps of coal in a stocking, there would be no need for testers.

That is why I recommend that testers buy a gift for the worst Developers on the team. I say the worst, because it is these venerable engineers that make necessary most of the testing we do. You don't need to reward the good developers - if they are not going to send some wonderful bugs your way, you don't need to get them a thing. So Karen, Jeff, Dominick, and Alex - no gifts for you because you make me work too hard to find those precious few bugs.

But what is the best gift to show how much you appreciate the Bad Developers and their lack of best practices, unit testing, common sense, and skill than a fruit-cake. Nothing says "Thanks for the lousy code! Keep up the bad work!" like a cake made of fruit. This classic no-prize is just what every Bad Developer needs and wants.

Sharpen the Flaw

The Seven Habits of Highly Effective Testers
Part 7 of 7

This is finally the seventh post in a series many thought would never be finished. It is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

Like college freshmen, bugs rarely live alone. A group of bugs living together is called a pod, or maybe that is dolphins. Swarm? Colony? The best name would have to be the same as crows - a murder of bugs. Yes, I realize I am getting off track. Sorry.

The point is, when you find one bug, you can be sure that there are other bugs nearby. As a tester, you should exploit this by banging on the features in which you find the bug. Like pulling on a loose thread, testing heavily around a new bug can lead to the whole program unravelling.

You should also test similar functions in other features. Developers are at least as lazy as normal people, so when they can reuse a bit of nifty code, they will, and thus make another instance of the bug. The right way for developers to do this is to abstract the function and then call it from everywhere that needs it, but as we know, the right way to do things is not always the way things are done. And thank goodness for that.

Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind
3 - Put Worst Things First
4 - Think When/When
5- Seek First to Understand, Then to Exploit
6 - Sinner Guise

Sinner Guise

The Seven Habits of Highly Effective Testers
Part 6 of 7

This is the sixth post in a recently neglected series which is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.

If Moses was alive today he would be a technical writer. Just like the ten commandments, the purpose of software documentation is to tell people what they should do and should not do. Also like the ten commandments, much software documentation is ignored.

Software is designed to be used in a certain way. Careful use cases and flows are thought out and implemented by designers and developers. Testers test these happy flows and make sure everything works the way it should. All is well until a user gets the software and tries to do something with it that no one ever imagined. What happens next is crucial.

A sign of quality software is that it either handles the abuse, or at worst, fails gracefully. It is our job as testers to make see that this happens. To do this, part of your testing should include ignoring everything you know about what a user Shalt Do or Shalt Not Do, and go crazy:

• Put negative numbers into text fields
• Put ridiculously long strings anywhere you can,
• Paste inappropriate stuff - like binary files into text boxes
• Open 13 instances of your app at once
• Unplug the network or the power in the middle of an important flow
• Change the system date and time
• Install on a system way below the minimum specs
• Use weird characters like ì È Ö
• Run 10 other apps while you are testing
• Delete crucial files
• Don't close other applications before proceeding

In other words - be great and terrible in the way you treat the software. Put on the guise of a sinner and break all the commandments. The developers may not forgive you for doing this, but your users won't forgive if you don't.

Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind
3 - Put Worst Things First
4 - Think When/When
5- Seek First to Understand, Then to Exploit

Next Up:
7- Sharpen the Flaw

Firefox Beta Goes To 11 (Thousand)

When I got the heads-up from Josh and Duncan that the beta of Firefox 3 was now available, what put a twinkle in my eye was that Mozilla has fixed 11,000 issues with this release. I could not believe it, but here is the info directly from the source:

Firefox 3 Beta 1 is based on the new Gecko 1.9 Web rendering platform, which has been under development for the past 27 months and includes nearly 2 million lines of code changes, fixing more than 11,000 issues.

My first reaction was "How great and wonderful Mozilla is for giving us such a vastly improved product!" This was followed 2 seconds later by "Yikes! My most used application and the product my company has hitched our wagon to has at least 11,000 known issues!"

My next thought was "How great and wonderful the Mozilla test team must be for finding that many bugs and driving them to resolution." I doubt I've found anywhere near 11,000 bugs in my career, and I've been a software tester since long it was cool and hip. Even if these issues have been accumulating over the last 27 months, that is more than 10 bugs a day - and that is just the ones they fixed.

I then came to the conclusion that Mozilla actually has hundreds of millions of the best kind of testers - free ones - aka users. That is the only way they could find, track, and fix 11,000 bugs - with a strong and active community. So, my final thought was "How great and wonderful the Mozilla community is for finding and reporting all these bugs!" Mozilla should buy them 11,000 beers.

More on the Firefox 3 beta on Techmeme.

A Touch of Quality

I am not, by any means, an Apple fanboy. For most of my life I've been quite the opposite. A couple of years ago, I got a first generation iPod Nano, and thought it was fairly decent. Last year I got a MacBook and it has slowly become my laptop of choice - especially when I travel. When the iPhone came out, I was intrigued, even more so after getting to play with one. Since I am boycotting Cingular however, the iPhone would have to remain something I would admire from a distance.

So, you can imagine my interest when Mr. Steve announced the iPod Touch which had many of best parts of the iPhone but with no phone, and therefore no Cingular (now AT&T) contract. I resisted temptation for what seemed like a long time, but a few weeks ago I gave in and bought a Touch. I am heavy into gadgetry, so it really means something when I say that the iPod Touch is the

Best Gadget Ever.

This is a blog about quality, and I am here to tell you that the Touch is chock full of quality, both in the hardware and in the software. Lets start with the hardware. The thing is crazy thin. Even when I was 6'3" and 140 pounds (a long time ago) I was nowhere near this thin. Even so it feels solid and sturdy. The screen is plenty big and bright, and easy to read. The wi-fi is strong. Even though there are only 2 buttons, this is actually just enough, and I have yet to find myself wishing for more.

As good as the hardware is, the software is even better. Whether you are making your way through music, videos, pictures, or real web pages the navigation is thoughtful and intuitive. Even entering text is not that bad. The software is also very solid - and this from a guy who makes his living breaking software.

That is not to say the iPod Touch could not be improved. Some changes I'd like to see:

• A speaker - This thing begs to shown off and passed around, and so it would be real nice to have a speaker to play the music and video soundtracks so your friends can enjoy it without getting goo on your ear buds.


• Some games - These will be coming after Apple opens up the platform next year, but it would nice if they included some simple time killers from the get-go


• Attached mode - This is a bit unusual, but stay with me. Most of my iTunes listening occurs on my computer - the screen for which is taken up with stuff other than iTunes. Wouldn't it be cool if I could plug in my Touch, then use it as dedicated interface to control iTunes on my computer?

Good work Apple. The Touch is so good that I may actually break my Cingular boycott and get myself an iPhone one of these days - though I am going to *try* to wait until the next generation which I *hope* will be announced in January.