Product release reflections

We released a major update to our Milestone Tracking Matrix product this evening.  When release milestones are achieved it is important to reflect upon the good points as well as the areas where improvements are needed.  I’m tired tonight so I’ll simply share with you my scorecard for this release by including the e-mail I shared with the entire company:

“

   I wanted to take a moment and acknowledge one of the largest product releases we have ever delivered.  We met more than 30 customer-driven requirements, some small, some very large.  Not only did we meet these requirements but we also revamped the entire user interface so as to be much cleaner and easier to use, we delivered.  Our customers will benefit greatly.

   He is my quick scorecard:

• Customer Requirements: A-.  We met the vast majority of requirements, only falling short on some minor points.
• Scheduling: C.  While I take ownership for this one we must all seek to become more rigorous about upfront scheduling, tracking progress, and communicating challenges.
• Code Design Quality:  B.  We made some solid improvements in terms of code maintainability.
• Test Coverage. B.  We did a really good job but need to look at how to do a better job allocating resources earlier.  The challenge here is both schedule-based and resource limitations.
• User Interface Design: A.  We should be really proud of the work in this area, it feels like a brand new application.
• Usability:  Good usability improvements, customers will notice the changes we have made.
• Performance Improvements: C.  We only made minor performance improvements in most areas.
• Scalability: C.  We only made minor improvements.

Overall Score:  B

A great product release overall. We have plenty of room to get better, which is good….  If we did it perfectly what would we have to look forward to?

“

How does this compare to most of your releases?

John

Response to comments on UI automation of web applications

I was happy to receive a very well thought out response to my recent post (thank you Martin) on the lack of ROI for developing GUI automation for web applications.  Instead of burying the response in the comment section, I have posted the poster’s comments and interspersed my thoughts as well.

Here are Martin’s comments with my responses in quotes throughout.

——————————————————————————— 

Unfortunately, your blog entry does not mention how automation was done in either of the mentioned cases. It also does not seem to consider that test automation often is or should be aimed at regression testing (finding new issues in existing functionality that was changed). But I would like to take a step back for a moment.

John’s first thoughts:
“
Automation always helps to reduce the time to complete regression testing, and portions of your automation should also be run as part of your regular build process.  The important point, though, is that you can build solid automation focused on at other levels (web services, unit tests, etc..) that will be far more reliable to run and maintain, providing huge value without the same maintenance requirements as you will find with GUI-based automation.

I have been building web applications for years and know that the above types of automation find the majority of the bugs.  You will, of course, find a large #s of bugs is in cross platform, cross browser, tests cases.  For example, you will often find bugs on the Mac with Safari that you do not see on Windows XP with IE 7.  The problem, however, is finding GUI automation tools that are capable of running across the platforms you must support.  If you can find, or build one, you must often build additional complexity into your test automation framework, leading to larger maintenance headaches.  I still contend that these types of issues can be found more cheaply by manual resources.
“
Let us consider what factors affect the ROI of a test automation project, for functional testing. Actually, the kind of interface (web, GUI, messaging, whatever) is not the essence, so the below would seem to apply to all test automation.

First of all, how do you define ROI? In terms of money only? That would depend on why you are automating in the first place. What business objectives is it meant to support? Lower cost? A common objective, certainly. But shortening time to market, higher quality, a better grip on the test process, and more effective use of scarce resources can also play their roles. Often a combination of these is desired, and sometimes cost is considered rather less important than others. But to keep it simple, I will focus on cost here.

John’s response:

“Businesses must always focus on ROI in terms of money, both short term expenses as well as long-term investments which include everything from maintenance costs to turnover of employees that are not happy with their career growth.  While I will not go into a thorough multi-page analysis of the ROI, consider the following:

• Ignore the cost of the tools.  10 years ago these tools were expensive and they no longer are.
•  The yearly salary of automation engineers is significantly higher than manual testers.  My preference would be to have a smaller number of automation engineers focused on unit testing, web services testing (more expensive, I agree, than GUI automation engineers) with a few more manual testers. In simple terms I can replace every GUI automation engineer with two manual testers, a good trade-off for test coverage and for our economy with more workers working.
• From a career growth perspective, QA engineers have a better career path because those doing automation are working with languages like Java and C# as opposed to tool-specific languages (like SilkTest 4Test as an example).
• The cost for each business will vary from company to company.  However, analyze the bugs in your product for the last year and ask yourself:
  • What % of bugs would have been found with unit or web services testing?
  • What % of your application bugs were platform specific?  With those bugs would a manual tester have found the other UI bugs as well?  You need to invest in testing the other platforms anyway and you cannot do it with most functional automation tools.
  • How many hours per week do you spend in maintaining your automation?
  • When you run your entire automation suite, how many hours of manual testing are you replacing it with?
  • For the location that you are hiring at, do the skills exist to hire/replace existing people will needed?

There are clearly more questions but those listed above are the most critical.”
Also, it is not so much the extent to which the testing is automated that matters. Dorothy Graham says it could make sense even to automate just 2% of your testing, and I would agree. The question is whether the investment pays off, not how much you invest or how much you automate. An for a pilot project, trying out automation on 2% could also be a good idea.

I think what determines ROI is (mainly) the following:
- The setup cost (licenses, perhaps servers, training),
- the cost of creating the testware (automated test + supporting framework / software),
- the cost of maintaining the testware, and
- the cost reduction in manual testing.

John’s response:

“
You are right, of course, regarding the fact that you do not need to automate 100% of the test cases.  In fact, you would fail if you tried to automate all test cases, but that’s another discussion entirely.”
The setup cost depends on whether you buy or build, whether you use open source or commercial tools, etc. More and more people are finding out that free software can offer a lot of bang for no bucks (with no vendor lock in …).

The time and effort spent on creating the testware is interesting, but usually much less than the time and effort spent on maintaining it, as with all software. So let us consider the total effort. It depends to a large extent on:
- The development process (e.g.: how much time between releases),
- The product planning (e.g.: how much change per release),
- The test automation approach (an advanced one using a DSTL requires much less maintenance than Record & Playback; is automation complete when the product is ready for testing or is that when automation can start – DSTL=Domain Specific Test Language), and
- The skill of the automator (the quality of the DSTL, the maintenance sensitivity and maintainability of the testware).

The cost reduction I will leave for you to determine.
The amount of time between releases does not have to be an issue if the amount of manual maintenance is low enough. A good DSTL (high level, no irrelevant execution details, tooling details and interfacing details in the test) helps a lot by reducing the maintenance to the test. A good automator helps a lot by reducing the maintenance to the rest of the testware. The page object concept, for example, can be a great help here, both for web and GUI apps.

John’s response:

“
Again, good points.  However, the number of great automators vs. average automators ensures you never achieve the ideal situation above.  Even if you do, however, investing in other forms of automation combined with manual testing against the web front end is almost always cheaper.
“
One guy’s research indicated that about two out of every three test automation projects fails sooner or later. A major factor is that many are not aware that test automation is software engineering, and people with the wrong skill set do it. As a result, they may work hard, but not very smart. It is a bit like the guy who is asked what he would do to empty a bath tub when given a tea spoon, a regular spoon and a really big spoon. When he choses the big spoon to be able to work faster, he is asked why he does not simply pull the plug …

Of course, there are limits even to what a skilled automator can do. But in my experience of 10+ years of creating solutions for automated testing of all sorts of systems, maintenance was never a problem.
John’s response:

“
While I don’t know who this ‘one guy’ is, the numbers make sense.  Software projects fail at a high rate, a rate that is sometimes overblown due to the lack of a solid definition of project success versus project failure.  Automation development, if you ignore simple record/playback, is software development.

I am fine with agreeing to disagree but wanted to thank you for the detailed and passionate response.  Passion is what makes life great, don’t lose it.
“

Functional UI automation for web applications does not make sense

More heresy, I know.  However, after spending nearly two decades building and testing software I know that it is the undeniable truth. Please understand the following:

• I am only discussing web applications. My logic will make more sense in a few seconds.
• All other forms of automation both make sense, and are critical, for web applications.  This includes unit testing, scalability testing, performance testing.

My reason for being against functional ui automation is simple, the ROI IS NOT there.  Let me tell you a couple of stories that I have seen, and heard, in recent years:

• In one case a company had one automation engineer.  Due to the rapid pace of code change associated with iterative development the engineer spent as much time updating the automation as they did running and analyzing results.  Since frequent builds were being run the automation was often not executed for a few days as the engineer fixed it and analyzed results.  Bugs found were sometimes no longer in the code or, worse yet, had already been found by manual testers.
• In another case I have saw a company invest in a larger automation team, with one group focused on new development and one team focused on maintenance.  The total coverage of the product would have, in my opinion, been improved if they had simply switched to a higher number of manual testers with a smaller number of automation engineers focused on performance and scalability.

The argument is always made that the automated UI tests you build will save you time throughout the testing cycle.  It does not do so with web applications.

The argument is always made that these tests will cover the “boring” areas that people will not run through on their own.  Again, I disagree.  Important bugs in the UI will be noted by the manual testers.  Bugs in subtle areas (such as your security model) should be caught by unit tests.

Am I right?  Have I missed anything that would make this argument invalid?  Let me know what you think.  You can find additional commentary here.

John

Why is it that the best bugs are found at the end of a release?

It never fails.  I have worked with QA groups ranging from 1 to 100 and you always end up finding the most interesting, and sometimes the most critical, issues right at the end of a product release.   There are a number of reasons for this phenomenon, here are the ones I have seen most often:

• QA does not receive working features until the end of the release.  Release dates are fixed and everything comes together at the last minute.  We have all seen this and it always results in two things:  A mad scramble by all involved; A patch soon after the initial release.
• QA is properly staffed and works on “proper procedures”, developing robust test plans for each feature.
  • This is great, but sometimes leads to the basic aspects of features being tested with limited to no real world testing taking place.
  • Your QA team should be building test scenarios from the original business requirements and testing these as soon as they can.  If the customer can actually accomplish the the business tasks you were trying to automate they will be more forgiving of typos in a dialog.  However, they will not care that the product is free of typos if the core requirements have not been met.
  • You must build in time for random or freeform testing throughout the cycle.  Let your team be creative.
• Lack of customer testing.   QA teams do great but they are not your final customers.  Supplement developer and QA testing with customer beta testing early in the process.
• Unfortunately, many times, you also fail to test for performance, scalability, and security issues until near the end.  Ensure this is done throughout the cycle or you will have even larger issues to deal with late, when you least want to make these detailed changes.

John

Unit testing, for better and for worse

For the sake of this post, unit testing is defined as testing the individual functions and code routines that make up your application.  You might be doing this manually or via automated tools.  The code you are writing might be part of the command console for a NASA mission, a stored procedure, or test automation.  Now that I have gotten those points out of the way…

You have to unit test everything you write.  It’s that simple.  Nothing, not even a simple one character change should be checked in or deployed without taking time to unit test the changes.  I’ve seen one line changes result in infinite loops taking down a web server in the middle of the day; it’s not pretty. 

How do I manually unit test my code?

It’s fairly straight-forward:

• Walk through every line of code in the debugger.
• Make sure your code is being called.  Early on in my career I once spent hours trying to figure out why a piece of code was not working, only to discover it was never being called.  That was a rookie mistake, don’t make it yourself.
• Examine the variables that are being set to ensure they are set as expected.
• Use watches to track variables that are not directly in the flow of your code.  Depending on the type of application you are working on they may be changing underneath your feet.
• If you have conditional aspects of your code, manipulate variables to ensure you hit all of your code paths.
• Make sure that callers of your function are able to work with your new routine or with the way you’ve modified an existing routine.  It’s easy to inadvertently introduce side-effects down stream and you need to be looking for unplanned changes.

How do I programmatically unit test my code?

 It’s time consuming to run through your code changes and it is easy to miss things if you are making changes to code that has been around for awhile.  Just as QA teams build automated test to ensure that the important, but tedious, test cases are covered on a regular basis, it is in the best interest of the development team to automate their own unit testing.   If you are going to automate unit test, here are some recommendations to ensure you are successful:

• Have a plan.  Define the requirements for what you will automate first.
• Set goals and define a schedule.  In one company I tried to keep the goal flexible, something along the lines of write a couple of unit test a week.  We rarely, if ever, met that goal.  Assign the task and measure it like any other project.  If you do not treat it as a priority it will not get done.
• Focus on calling those functions that scare you the most.  We all have those routines that exist in the depths of our code that no one dares to touch because if we so much as look at that code it will not ever compile again.  Automate those early.
• Don’t try to automate all of your unit testing.  Make sure your developers understand that they are responsible for the quality of the code they check in and that they need to test it manually before it’s checked in.
  • Some people do advocate test driven development (TDD) or automating the test after the coding is done and prior to check-in.   This is even better.
• Tie the unit tests into your builds.  Everytime you produce a new build of your code have the unit tests run. 
• If anything breaks put the team on alert and ensure the person that broke the unit tests is on the hook for fixing them.    Enforce this.  Keep your tests and build running cleanly and you will all be happier for it.

Have I missed anything that’s important inside your organization?

 John

SharePoint in Action: Test Management

This post is a follow-up to my SharePoint In Action overview post which can be found at SharePoint In Action: An Overview.  If you are interested in reading the other articles in this series search for “SharePoint in Action” on this blog.

In most startups Quality Assurance (QA) is one of the last roles filled.    I have worked in three straight startups and I have been fortunate to have two of the three fully buy into my view of QA.  QA is about more than just banging away at the keyboard to see what blows up.  Careful planning combined with an appreciation for how user’s interact with the system is critical.  Planning, execution, and accurate, unbiased, tracking and reporting will make a good team great.

As many of you know, I work at Swimfishand I am working hard to fully leverage our investment in Microsoft SharePoint.  As I began discussing my reporting needs with the QA group they quickly leapt at the chance to use SharePoint for their Test Management System, understanding that:

• We are heavily invested in SharePoint.  Using SharePoint for Test Management builds upon this investment and further leverages the knowledge the team has with the platform.
• As we grow we can more easily tie Test Management  metrics into our Project Management System(also in SharePoint).
• It’s out of the box templates, rich workflow, and powerful APIs make it a system we can fully customize for all of our needs.

To be honest, I was somewhat skeptical about using SharePoint for this task.  There are dozens of Test Management Systems available that would have immediately met our needs.  These systems do Test Management and they do it well.  Why create a new system?  Well, it came down to the three points above combined with the desire to let the team channel their ideas to solving the problem I had given them.   SharePoint provides the basic capabilities and for little extra work I can get the basic reports I need.  Forcing an alternative system down the team’s throat was not necessary.  Management 101, give the team a goal and then let them solve it. 

Constructing the Test Management Site

We created a “Budgeting and Tracking Multiple Systems” template by:

• Within Sharepoint we used the Site Actions menu and selected Create.
• We then chose “Sites and Workspaces”.
• Fill in the information on the form and choose the “Budgeting and Tracking Multiple Projects” template under the Applications Template tab.
• Press Create.  After a few seconds you now have your Test Management Site.

Using the system

The QA group was able to easily replicate the projects from our Project Management site and tie the test cases to these projects.  Fields have been added to track if test cases passed or failed, defects logged by running the test case, and expected vs. actual results.

I added a Management Overview view to track % completion, budgeted days, and combined this with reporting from Bugzilla to get an accurate picture of the current project from the test team’s perspective.

Upcoming challenges, future changes

People familiar with Test Management Systems can already see many of the challenges that we need to overcome.  None are large and I remain confident that the team’s decision to go with SharePoint will lead to a great integrated solution.  However, here are some of the items the team will have to overcome in the near future:

• We will need a way to easily replicate test cases from one project to another project when they are both associated with one core product.  For example, we released Version 3.0 of our Mobile CRM producta couple of months ago and Version 3.01 is set to be released very soon.  It has to be easy to re-use testcases from 3.0 in Version 3.01.
• As the team identifies bug through their test process I would like it to be easy to cleanly tie the bug from the bug system (BugZilla) and the Test Management System.  Not hard, but not yet done.
• As developers fix bugs I want this to flow out to the Test Management system as part of the code check-in process (SubVersion). 
• If a project under test is a services effort associated with an engagement in our CRM system I want the status for the project to cleanly flow back to our CRM system.

Did the team make the right choice?  What else is missing in your opinion?

John

Should you “throw it over the wall” or implement QA hand-offs?

As developers we often feel that our code is of high quality, that we’ve nailed the requirements, that we have unit tested to the best of our abilities.  We’re often wrong.

When I worked at Lotus, dozens of years ago, we once had a team that was convinced that there was no need for a test team, that developers following good coding practices and building robust unit tests could produce the right product, on the right schedule, at the right quality.  This team believed in the nonsense that I discussed in this post.  As I recall, the team never shipped a new product (after more than 3 years) and the team’s management was replaced multiple times.

In one of my recent roles I had a boss who felt that many of the QA-related processes that I put in place were overkill.  In his opinion we were generally better off coding quickly, throwing it over the wall to a formal, or informal test team.  Coding as quickly as possible without the overhead of too much QA time, was key.  I failed to convince this person of the error of their ways.    The end result was slower than desired release schedules at lower overall quality and a general dissatisfaction with what we were building. 

The right answer, which I implemented at Brainshark and that we are evolving towards at Swimfish, is the QA hand-off.

• What is it?  When a developer believes they are code-complete for some piece of new development they sit down with the QA person, and documentation writer, responsible for that feature or component.
  • They should walk through the feature and demonstrate that each requirement has been met.
  • They should raise any areas of concern they have with the feature.  Even after unit testing a developer may still have concerns that they need an additional set of eyes to review.
• What isn’t it?
  • It is not a technical design review.  While it is okay to discuss why technical choices led to certain areas of concern it is not a time to reconsider the approach.   It is the time to ensure that teammates understand what has been built so that they can perform system, performance, scalability, and other types of testing.
• What are the keys to success for the QA hand-off?
  • The developer should review the requirements prior to the QA hand-off.  It is not acceptable to come to this meeting without knowing if  you are “done”.
  • The developer should be done with all unit testing prior to the QA hand-off.
  • Documentation.  If appropriate, a one pager that describes any concerns the developer has with the feature.
  • Working code on the developer machine so that the developer can walk the teammates through the feature.
  • QA and Documentation teammates should come prepared, having an understanding of the feature requirements, not learning through the course of the meeting.
  • Defects will often be discovered through this meeting.  The developer should fix all of these items before moving onto the next feature.

Would this work for your organization?  Do you already do something similar?  Let me know what you think.

John

Why waste time testing, developers can deliver 0 bugs…

While I have had this conversation at nearly every company I have worked at, I do not believe that testing is a waste of time nor do I believe that you can deliver 100% bug free code ever.    I recently saw this great post which got me thinking:  Top ten most infamous software bugs of all time.

• Yes, you can deliver software, even Enterpise software with no QA.  QA is all about risk management, helping you to better understand what you are about to ship, where the risk areas are and gives you a certain sense of comfort.  You can go without this if money is tight but there is major risk with this approach.
• Developers should just write code with 0 bugs.   This is ideal of course, and, perhaps even feasible when you look at the world in the micro level of individual functions.  Unit testing, of course, should help provide a solid level of comfort here but it is only a piece of the puzzle.  The bugs that will bite you hardest come from the interplay of functions, from incomplete understanding of requirements, from poor performance.  Individual developers focused on individual features will not catch these.
• Unit testing is the answer, get 100% code coverage and dump the QA team.  Same basic problems as noted above.  Good unit testing takes time and you are often spending higher $$s on the developer resources than on the QA resources to get the same level of coverage.  There is definitely an argument that automation is worthwhile, which I completely agree with.  However, it comes down to managing expenses and in most startup environments it’s cheaper and nearly as cost efficient to have a good QA team on board than to have your developers writing large amount of unit tests.

Thankfully, I have not had this discussion at my current company.  We have a good QA team and developers that are investing a proper amount of time on their unit testing.  We are also striving to live by my Engineering Tenants which keeps us focused on doing the right things.
Let me know what you think. Have you ever worked in an organization successfully delivering software that did not have a QA team?

-John