Cache Test Dummy

Money may be the root of all evil, but it is cache that causes the biggest headaches when you are testing an app with any web based components.

Life in the fast lane of web development usually includes rapid cycles of development, deployment, and test. Cache is the monkey-wrench to this fine tuned machine because of the mysterious way that the browser locally stores files that would otherwise be served fresh off the server

I don't know how many times I've found a really good bug and excitedly reported it to the erstwhile developer only to have them say "Hmm, that's odd. Doesn't happen for me. Try clearing your cache." More often than not, this makes the bug go away, and makes me want to shoot myself.

The real problem is, testers are not the only ones that have cache. Users do too. And users don't tend to feel the joy of bugs likes testers do. What can be done? For one, you can ask your users to clear their cache. We just did that on our blog to help ensure our users and ourselves that our product is not to blame for issues some people are seeing in Firefox.

That is not the most elegant of solutions. It is better to find and fix these issues before a user ever sees them. So, when you find a bug that is caused by something being cached or not cached as expected, clearing your cache to make the bug go away is not enough. You need to go the next step of figuring out in what scenarios caching is causing trouble, then working with development to make them understand the situation, and improving the code to deal with it.

Because of the nebulous nature of how browsers handle cache, tackling these problems can be like driving into a wall at 65 MPH, but such is our role as testers. Better us than the users.

The Seen and Unseen

I found the kind of bug this morning that I really enjoy. Well, it would be more accurate to say I missed the kind of bug I really enjoy, but found it eventually.

Yesterday we updated some of the images in our product. These images reside on our server, and are dynamically loaded when a user opens the options page in their browser. When the new images were put on the server, I made sure they were right in both the released version (since those users will see them too) and in the upcoming version that I am currently testing. Good job testing by me, right?

Wrong.

Where I did not test them was on a Mac. This was a simple change and I did not think it was necessary, especially given all the other stuff that I was testing so we could finish this release on time. My mistake. I got a midnight e-mail from my boss that there was a small white space on the bottom of two of the buttons on the Mac:

OK, not a huge deal, but something we wanted to fix right away since all our current users (who use Macs) could see this. This is where the story gets good. New graphics were created and uploaded to the server this morning. Both my boss and I checked them on Windows and Mac, and the white space was definitely gone. I gave a big thumbs-up to the change. Good job testing by me, right?

Wrong.

About ten minutes later, in the course of testing something else, I realized that in fixing the whitespace problem, somehow the names of the images got switched for the first two buttons. In other words, the first button looked like the second button should, and vice versa, and so neither one did what it said it would. This was a much worse problem, and not just because it effected Windows users too.

What are the lesson to learn from this? I'll save the testing on Mac vs PC lessons for another post (or ten). I'm also going to save the lesson about how fixing minor bugs often creates major bugs. Today's lesson is about how a software test professional such as myself (not to mention my boss) could miss the blatant fact that the buttons were totally wrong when we were looking right at them. It was because of selective attention - we were focused on whether or not there was a little white space under the buttons. In software testing, focus is a fickle mistress. You can just as easily have too much as you can not enough. That is why I like exploratory testing. By not looking for just one specific thing, you tend to see a lot more. But that is also the topic for another post.