It never fails. I have worked with QA groups ranging from 1 to 100 and you always end up finding the most interesting, and sometimes the most critical, issues right at the end of a product release. There are a number of reasons for this phenomenon, here are the ones I have seen most often:
- QA does not receive working features until the end of the release. Release dates are fixed and everything comes together at the last minute. We have all seen this and it always results in two things: A mad scramble by all involved; A patch soon after the initial release.
- QA is properly staffed and works on “proper procedures”, developing robust test plans for each feature.
- This is great, but sometimes leads to the basic aspects of features being tested with limited to no real world testing taking place.
- Your QA team should be building test scenarios from the original business requirements and testing these as soon as they can. If the customer can actually accomplish the the business tasks you were trying to automate they will be more forgiving of typos in a dialog. However, they will not care that the product is free of typos if the core requirements have not been met.
- You must build in time for random or freeform testing throughout the cycle. Let your team be creative.
- Lack of customer testing. QA teams do great but they are not your final customers. Supplement developer and QA testing with customer beta testing early in the process.
- Unfortunately, many times, you also fail to test for performance, scalability, and security issues until near the end. Ensure this is done throughout the cycle or you will have even larger issues to deal with late, when you least want to make these detailed changes.
John
September 29, 2010 at 3:01 pm
John – Are product companies/product managers using their Social Networking for product feedback or beta testing. Are they using business networks (LinkedIN) or even social networks (Facebook) to have people try features? Have you tried it? Does it break any “unwritten rules” of what social networks should be use for? Very interested in your thoughts or results if you have tried it. – GF
September 29, 2010 at 3:25 pm
It breaks no unwritten rules and does, in fact, provide a richer interaction with customers (beta testers and others). One of two approaches that I have seen works best:
- Create a separate Facebook page for your beta testers and invite them in, share feedback, tips, etc… Give them something in return for their time, of course.
- Take your company’s LinkedIn Group and create a subgroup for the same purpose.
Note, however, that both approaches work IF you have setup your social media program ahead of time and created a value-driven community in either space. If you try a community-driven approach without first creating the community you will fail 95%+ of the time.
If you could use help with your social media approach, let me know.
John
May 6, 2009 at 1:30 am
Great question. Lots to think about.
My comments at http://sethmorris.spaces.live.com/blog/cns!51B088CB629542CE!228.entry
April 28, 2009 at 12:08 am
Nice posts. Some suggested reframes:
QA does not receive working features until the end of the release.
Testers do not receive features that the programmers believe might work until well beyond the time that they optimistically figured they might have them working. Can this really be a surprise to anyone, though?
QA is properly staffed and works on “proper procedures”
If testers are working on “proper procedures” that don’t allow the most rapid and the most effective feedback, are the procedures really proper? Or are they merely idealistic?
If the customer can actually accomplish the the business tasks you were trying to automate…
You’re never trying to automate a business task; you’re always trying to use automation to enhance or accelerate or extend a business task. The distinction is, in my view, significant.
—Michael B.
April 27, 2009 at 4:46 pm
Testing is like brain storming.
We encounter & address one problem. This can lead us to ask:
“Are there any similar problems in the new software?” (where a variation on the kind of error or mistake that led to this.)
“Does this kind of problem exist in other software that we’re responsible for?” (other software had the same kind of mess-up, that we may not have previously stumbled over.)
“What kind of flaw in software development can lead to this type of problem even coming into existence?”
With brain storming, one idea leads to another.
There can be dumb ideas that lead to great ideas.
The ideas we abandon along the path are sometimes called “bridge ideas.”
Same concepts can apply to testing.
With proper documentation of problem encountered by whom & how to replicate it, type of problem (whose responsibility to fix), claim that it has been fixed, whose responsibility to test that, claim that it was tested & now is resolved … people reviewing the lists of bugs, fixes, etc. can be inspired to ask similar questions about whether we have similar problems, that ought to be checked out.
April 27, 2009 at 1:40 pm
[...] Why is it that the best bugs are found at the end of the release? [...]
April 22, 2009 at 11:59 pm
The Use Cases developed during the requirements phase should be a collaborative process with QA. Ad-hoc testers are sharks that look for everything and can help elicit requirements. Having them involved early also conveys acceptable errors (“We don’t care if the phone number is formatted correctly or not.”) and unacceptable errors (“The phone number must be entered in this format: (###)###-####.”)
Cliff Adams
http://adamsenterprises.wordpress.com
http://www.adamsenterprises.com
April 22, 2009 at 6:42 pm
@Erin Ivers-Holman : I completely agree with the human aspect. Testers are first Humans… The Procrastination.
Now on to the “logical” reason…
Being in an Agile team, Most of my lateral thinking may go unnoticed as that has already helped my development team in building the right product.
And this lateral thinking if applied later, would have resulted in the “Best Bugs” found later in the cycle.
So, I end up with “no Best Bugs”
, but a good product early in the cycle.
That is the funda of being Agile. Saving the cost of the company.
BE AGILE… IT HELPS!!!
April 22, 2009 at 1:05 am
A few comments.
1) The more you fix the more you break
2) The longer the tester gets familiar with the program the better they get at finding problems
3) Some problems can’t be seen until all of the parts are in place
4) Requirements creep or removals might enter the picture
5) Intermittant errors show up better with a full loaded system
6) Requirements are never complete. Once every thing is in place a good tester should be in the gray areas just beyond the limits of the requirements.
7) Another problem at this stage is to convince development to that problems exist without requirements.
April 21, 2009 at 10:40 pm
sometimes the most interesting bugs are *introduced* near the end of the cycle. If you’re in serious crunch time, sleep-deprived developers tend to do crazier things under stress than they would do otherwise.
April 21, 2009 at 4:10 pm
some more points on the same lines.
In addition to the points mentioned here..some more challenges that we came across during the test cycles are
=> Features creep: in during the development phase and the code undergo constant changes. And when the realease team realises about the schedule, we would have reached the later part of test cycle. The entire team jumps into the schedule part and start containing the code changes. This is when the test teams get to have a more stable build and start testing the features from the requirement point. Until then, test teams spend most of the effort in repeating the basic test cases. Unproductive effort.
=> The requirement analysis is lead by development teams most of the time. What the development teams think as the feasibility becomes the design. Though the design may have fundamental flaw from design per se, it gets accepted for want of time. The test teams are clueless as to how this is getting embraces and it takes time for them to understand the big picture…By thee, we would be at the end of release …all the way testing to get the build stable..
April 21, 2009 at 4:13 pm
Great comments from everyone. Also, this has sparked great conversation on the Linked In QA groups. Love a good topic.
John
April 21, 2009 at 7:16 am
I’d add one more – quite often during early stages of testing there’s plenty of easy-to-find bugs. These are submitted and fixed soon. Then testers have to to put more effort to find other glitches and they have less distractions (yes, submitting a bug to bug tracker is a distraction from testing) so they start finding all these hidden issues. Hard to track them down, hard to fix them, but yes, they bring the most fun to deal with them.
April 21, 2009 at 6:44 am
Apart from the above reasons, there are certain reasons according to me for these bugs.
a) At the start phase, when we test, bugs are found on the basic functionality, which sometimes block the rest of testing (of a particular module). When we get a fix, and test the module and flow in the application, we find some more bugs. Which sometimes comes at the end due to continuous bugs and fixing.
b) Sometimes the bugs occur as part of ‘fix’ i.e. the devs have fixed the code at one end, and that causes another issue to prop up. Since there are continuous builds, and testing takes place, by the end, when we do a complete Regression testing before a release to check no fixes are hurting anything, we uncover some interesting issues.
c) As i said in first, we need the system stable and when it is quite stable we tests for E2E test cases, which again provide insight into some of the issues (these issues might have been ther, but they can only be covered as part of E2E with real data). These also can come up due to some recent fixes.
So, most of the times, these are also cases where we see interesting bugs. We call them interesting for the mere fact that they were not so easy to detect in a normal functional testing. Maybe they were working initially, and later for a last minute fix, they propped up again (which we’ll see during our regression testing).
This is the reason, that many times even after a good testing, you find Production bugs, coz that’s one area which is difficult to simulate at the test environment also.
Cheers!!! Please correct me if i am wrong.:)
April 21, 2009 at 12:00 am
@John,
Reading your blog entry, I think you covered most the reasons for the “best bugs come last” phenomenon.
Let me just add one more:
In the mobile space, part of the ecosystem–the carrier channel–is not under the control of the mobile application developer. I have been involved in several situations where a feature worked until it was tested just before submission to the carrier, and it was due to last-minute (and unannounced) changes to the carrier’s data network (these were client/server applications). Hard to plan for this.
April 20, 2009 at 10:37 pm
The main cause behind this problem is the lack of clear understanding of the requirements in the development\Testing phase and the poor communication between development team and customer/stakeholder. At the time of release phase, which involves more customer interaction helps in better understanding of the requirements which could be contradicting with the existing requirements.