Wednesday, September 24, 2008
The Lucky Thirteen
Harry McCracken has done a great post on The Thirteen Greatest Error Messages of All Time. This baker's dozen rouge gallery should bring joy to any tester's heart.
Monday, September 8, 2008
Who Tests the Tester?
Finding the right tester to join your team is tricky business. For one, software testing is not something many people have degrees in. Instead it is something most people learn on the job, which is a bit of a chicken and egg situation in of itself. Secondly, it seems many people only visit the testing profession rather than making a career of it, so the people with the right experience might not be interested in the job.
One thing we do we selecting candidates for an open Test Engineer position is to go give a quiz. In its current form it is only six questions that we give people up to an hour to answer. This provides a good level-set for what kind of tester a person is likely to be. I won't share the exact questions here in case some savvy applicant finds this blog. I will however give this much info:
One thing we do we selecting candidates for an open Test Engineer position is to go give a quiz. In its current form it is only six questions that we give people up to an hour to answer. This provides a good level-set for what kind of tester a person is likely to be. I won't share the exact questions here in case some savvy applicant finds this blog. I will however give this much info:
- Some of the questions are about what test cases would you write to test some made up app
- Some of the questions deal with when it is best to use different styles of testing
- One of the questions is a trick question - it asks about how you can tell when you've found all the bugs in an app. No one has gotten this one right yet.
Wednesday, April 9, 2008
To Find the Unfindable Bug
This morning I have been tasked with reproducing an unreproducible bug.
Since the last update of our software, we've heard from a very small number of new users who were unable o create an account. This is a big deal for this small percentage of people. It is possible that the number of people that are reporting this problem are a fraction of the people who are experiencing it, since many people will just drop-off if they can't create an account, but even so, we also know that most people don't see this problem. We've also not seen this problem during extensive testing.
Except that we have seen this problem in testing. A few times in dozens and dozens of attempts. Each time, the exact steps were recreated, and it worked fine the next time. Its what we in the testing business call "a fluke".
But, back to the business at hand - how can I solidly recreate this most elusive of bugs? The first thing to do is to find out all we can from the people that experience the bug. Using our best detective skills, we should try to determine how the people who see the bug are different from those who don't. It could be something about about how they are interacting with our product or something about their environment (physical resources, other software, even their physical location). If you can isolate what the difference is, it should be easy to reproduce.
If you can't isolate the difference, then it is time to bring in your testing skills. Two approaches you can try:
Since the last update of our software, we've heard from a very small number of new users who were unable o create an account. This is a big deal for this small percentage of people. It is possible that the number of people that are reporting this problem are a fraction of the people who are experiencing it, since many people will just drop-off if they can't create an account, but even so, we also know that most people don't see this problem. We've also not seen this problem during extensive testing.
Except that we have seen this problem in testing. A few times in dozens and dozens of attempts. Each time, the exact steps were recreated, and it worked fine the next time. Its what we in the testing business call "a fluke".
But, back to the business at hand - how can I solidly recreate this most elusive of bugs? The first thing to do is to find out all we can from the people that experience the bug. Using our best detective skills, we should try to determine how the people who see the bug are different from those who don't. It could be something about about how they are interacting with our product or something about their environment (physical resources, other software, even their physical location). If you can isolate what the difference is, it should be easy to reproduce.
If you can't isolate the difference, then it is time to bring in your testing skills. Two approaches you can try:
- "The Crazy Uncle" - Test unusual user flows. Since most people don't see the bug, perhaps we need to try something weird and unexpected to make it occur. Never underestimate a user's ability to use your software in a way that is totally unimaginable.
- "The Big Snafu" - Make what can go wrong, go wrong. This means limiting or disabling your systems resources while going through the use case. This could include such things as unplugging from the network half way through, removing the install CD at the wrong time, or devoting but a sliver of your system's RAM to some application other than what you are testing.
- "Brute Force" - Test the usual flow many many many times. It could be what is happening is some sort of race condition that will only occur 1 out of 100 times. This approach is extremely time consuming and boring (and therefore a good candidate for automation), but if you can make it fail when everything else is normal, that is important information that the developer can use to fix it.
If after all this you still can't reproduce the bug, then your choices are to keep trying, live with it, or hope the developers can find it by scrutinizing and carefully reviewing the code. None of this is especially appealing, but no one promised you a pleasure cruise.
Does anyone else have suggestions on ways to find the unfindable bug? Please add them to the comments if you do.
Wednesday, March 26, 2008
One of Those Days
Some days you get in a zone where the bugs come out of hiding to follow you like you were the Pied Piper of Defects. For me, that describes exactly how this day wasn't.
One thing I was testing involved passing strings of text. Of course all the basic, simple text worked fine. I also tested such odd ball characters like brackets and quotes and other punctuation. I even tested non-English characters including € and the like, and even some Japanese characters. Everything I tried worked perfectly, so I declared it good to go.
My boss then decided to give it a try. He happened to choose some text including a $. Of course, it failed miserably.
Later I was testing a new widget. I tested in on FF on Windows, and found some issues. These were fixed. I tested it on IE7, found some issues, and these were fixed. I tested it on FF3 Beta on Mac, and it worked there. It was good to go. Or would have been, had it not been on Safari. Guess who tried on Safari? That's right - my boss.
Sigh.
Some days you get the bugs. Some days the bugs get you.
One thing I was testing involved passing strings of text. Of course all the basic, simple text worked fine. I also tested such odd ball characters like brackets and quotes and other punctuation. I even tested non-English characters including € and the like, and even some Japanese characters. Everything I tried worked perfectly, so I declared it good to go.
My boss then decided to give it a try. He happened to choose some text including a $. Of course, it failed miserably.
Later I was testing a new widget. I tested in on FF on Windows, and found some issues. These were fixed. I tested it on IE7, found some issues, and these were fixed. I tested it on FF3 Beta on Mac, and it worked there. It was good to go. Or would have been, had it not been on Safari. Guess who tried on Safari? That's right - my boss.
Sigh.
Some days you get the bugs. Some days the bugs get you.
Friday, December 21, 2007
Pay Back Bad Developers
Here we are, in the thick of the holiday season when we give gifts to the important people in our lives. When doing your Christmas shopping, you should be careful not to overlook the founders of the feast - the Bad Developers.
I say this not because developers write the code that gets sold to your users, thus paying the bills. I say this because without Bad Developers, and the wonderful bugs they tuck away in the product like lumps of coal in a stocking, there would be no need for testers.
That is why I recommend that testers buy a gift for the worst Developers on the team. I say the worst, because it is these venerable engineers that make necessary most of the testing we do. You don't need to reward the good developers - if they are not going to send some wonderful bugs your way, you don't need to get them a thing. So Karen, Jeff, Dominick, and Alex - no gifts for you because you make me work too hard to find those precious few bugs.
But what is the best gift to show how much you appreciate the Bad Developers and their lack of best practices, unit testing, common sense, and skill than a fruit-cake. Nothing says "Thanks for the lousy code! Keep up the bad work!" like a cake made of fruit. This classic no-prize is just what every Bad Developer needs and wants.
Wednesday, December 19, 2007
Sharpen the Flaw
The Seven Habits of Highly Effective Testers
Part 7 of 7
This is finally the seventh post in a series many thought would never be finished. It is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.
Like college freshmen, bugs rarely live alone. A group of bugs living together is called a pod, or maybe that is dolphins. Swarm? Colony? The best name would have to be the same as crows - a murder of bugs. Yes, I realize I am getting off track. Sorry.
The point is, when you find one bug, you can be sure that there are other bugs nearby. As a tester, you should exploit this by banging on the features in which you find the bug. Like pulling on a loose thread, testing heavily around a new bug can lead to the whole program unravelling.
You should also test similar functions in other features. Developers are at least as lazy as normal people, so when they can reuse a bit of nifty code, they will, and thus make another instance of the bug. The right way for developers to do this is to abstract the function and then call it from everywhere that needs it, but as we know, the right way to do things is not always the way things are done. And thank goodness for that.
Part 7 of 7
This is finally the seventh post in a series many thought would never be finished. It is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.
Like college freshmen, bugs rarely live alone. A group of bugs living together is called a pod, or maybe that is dolphins. Swarm? Colony? The best name would have to be the same as crows - a murder of bugs. Yes, I realize I am getting off track. Sorry.
The point is, when you find one bug, you can be sure that there are other bugs nearby. As a tester, you should exploit this by banging on the features in which you find the bug. Like pulling on a loose thread, testing heavily around a new bug can lead to the whole program unravelling.
You should also test similar functions in other features. Developers are at least as lazy as normal people, so when they can reuse a bit of nifty code, they will, and thus make another instance of the bug. The right way for developers to do this is to abstract the function and then call it from everywhere that needs it, but as we know, the right way to do things is not always the way things are done. And thank goodness for that.
Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind
3 - Put Worst Things First
4 - Think When/When
5- Seek First to Understand, Then to Exploit
6 - Sinner Guise
Tuesday, December 4, 2007
Sinner Guise
The Seven Habits of Highly Effective Testers
Part 6 of 7
This is the sixth post in a recently neglected series which is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.
If Moses was alive today he would be a technical writer. Just like the ten commandments, the purpose of software documentation is to tell people what they should do and should not do. Also like the ten commandments, much software documentation is ignored.
Software is designed to be used in a certain way. Careful use cases and flows are thought out and implemented by designers and developers. Testers test these happy flows and make sure everything works the way it should. All is well until a user gets the software and tries to do something with it that no one ever imagined. What happens next is crucial.
A sign of quality software is that it either handles the abuse, or at worst, fails gracefully. It is our job as testers to make sure that this happens. To do this, part of your testing should include ignoring everything you know about what a user Shalt Do or Shalt Not Do, and go crazy:
Part 6 of 7
This is the sixth post in a recently neglected series which is based ever-so loosely on Stephen R. Covey's book but focused on what you can do to be more effective as a software tester.
If Moses was alive today he would be a technical writer. Just like the ten commandments, the purpose of software documentation is to tell people what they should do and should not do. Also like the ten commandments, much software documentation is ignored.
Software is designed to be used in a certain way. Careful use cases and flows are thought out and implemented by designers and developers. Testers test these happy flows and make sure everything works the way it should. All is well until a user gets the software and tries to do something with it that no one ever imagined. What happens next is crucial.
A sign of quality software is that it either handles the abuse, or at worst, fails gracefully. It is our job as testers to make sure that this happens. To do this, part of your testing should include ignoring everything you know about what a user Shalt Do or Shalt Not Do, and go crazy:
- Put negative numbers into text fields
- Put ridiculously long strings anywhere you can,
- Paste inappropriate stuff - like binary files into text boxes
- Open 13 instances of your app at once
- Unplug the network or the power in the middle of an important flow
- Change the system date and time
- Install on a system way below the minimum specs
- Use weird characters like ì È Ö
- Run 10 other apps while you are testing
- Delete crucial files
- Don't close other applications before proceeding
In other words - be great and terrible in the way you treat the software. Put on the guise of a sinner and break all the commandments. The developers may not forgive you for doing this, but your users won't forgive if you don't.
Previous Posts:
1 - Be Protractive
2 - Begin With the Vend in Mind
3 - Put Worst Things First
4 - Think When/When
5- Seek First to Understand, Then to Exploit
Next Up:
7- Sharpen the Flaw
Subscribe to:
Posts (Atom)
