What feature? Some outsourcing thoughts…

At Swimfish we have been fortunate to have a lot of business, business that exceeds the capabilities of my engineering teams on the east and west coast of the US.  Of the last few months I have been outsourcing some of our development efforts and, while it has gone well in some areas, it has been frustrating in others.  I wanted to take a few minutes and share with you some of my experiences as well as tips that have helped us out.

Feature estimation

It is hard to accurately estimate, as I have written before.  However, beyond the normal due diligence that you should perform for any project estimates, I have found the need to pay even closer attention to our offshore projects.  I have seen the following, extreme examples:

• One project for fixing bugs that should have taken less than 40 hours was estimated at approximately 100 hours.  As I dug into the estimate I discovered a “standard” 40 hours of upfront design for development and QA, a “standard” used for most projects.   This “standard” was removed.
• A project that should take somewhere between 3 and 4 weeks to complete was estimated at roughly four times the expected size.  Some of this was due to larger than expected design time for both development and QA.  Pay attention to the details and avoid being surprised.

Missing requirements

Growing up I loved The Hardy Boys and other mystery books, in fact, I still do… However, I truly dislike mysteries that take place at work.  Here are a couple of the surprises I have encountered along the way:

• Requirements that were simply dropped or missed.  I have gone to product reviews where I had been told a set of features were complete only to discover that some percentage of the work was never done.  In some cases the items were missed, in most cases they were not understood and simply skipped.  Have frequent reviews and ensure that your product managers are working closely with the team to avoid misses.
• Bug fixes that were postponed because off-shore members felt that the bugs were not critical for the release.  Ensure that bugs are tracked in your central bug tracking system and that your regular reviews discuss the status of bugs.  Do not give up your authority when it comes to managing the quality of your products.

Clear requirements and specs

While your requirements and specifications will never be perfect, try to add enough visual information (such as screen shots) to clarify where language alone may not be enough.

Separation of QA and Dev

From the beginning of our offshoring relationship I have been trying to determine if it makes sense to leave both development and QA functions at the same off-shore company.    My initial thought is that it is a mistake to have both in the same company and to date, my worries appear to be playing out.  What do you think?

How does this compare to your experiences?  Do you have any stories to share?  Any tips?

John

Response to comments on UI automation of web applications

I was happy to receive a very well thought out response to my recent post (thank you Martin) on the lack of ROI for developing GUI automation for web applications.  Instead of burying the response in the comment section, I have posted the poster’s comments and interspersed my thoughts as well.

Here are Martin’s comments with my responses in quotes throughout.

——————————————————————————— 

Unfortunately, your blog entry does not mention how automation was done in either of the mentioned cases. It also does not seem to consider that test automation often is or should be aimed at regression testing (finding new issues in existing functionality that was changed). But I would like to take a step back for a moment.

John’s first thoughts:
“
Automation always helps to reduce the time to complete regression testing, and portions of your automation should also be run as part of your regular build process.  The important point, though, is that you can build solid automation focused on at other levels (web services, unit tests, etc..) that will be far more reliable to run and maintain, providing huge value without the same maintenance requirements as you will find with GUI-based automation.

I have been building web applications for years and know that the above types of automation find the majority of the bugs.  You will, of course, find a large #s of bugs is in cross platform, cross browser, tests cases.  For example, you will often find bugs on the Mac with Safari that you do not see on Windows XP with IE 7.  The problem, however, is finding GUI automation tools that are capable of running across the platforms you must support.  If you can find, or build one, you must often build additional complexity into your test automation framework, leading to larger maintenance headaches.  I still contend that these types of issues can be found more cheaply by manual resources.
“
Let us consider what factors affect the ROI of a test automation project, for functional testing. Actually, the kind of interface (web, GUI, messaging, whatever) is not the essence, so the below would seem to apply to all test automation.

First of all, how do you define ROI? In terms of money only? That would depend on why you are automating in the first place. What business objectives is it meant to support? Lower cost? A common objective, certainly. But shortening time to market, higher quality, a better grip on the test process, and more effective use of scarce resources can also play their roles. Often a combination of these is desired, and sometimes cost is considered rather less important than others. But to keep it simple, I will focus on cost here.

John’s response:

“Businesses must always focus on ROI in terms of money, both short term expenses as well as long-term investments which include everything from maintenance costs to turnover of employees that are not happy with their career growth.  While I will not go into a thorough multi-page analysis of the ROI, consider the following:

• Ignore the cost of the tools.  10 years ago these tools were expensive and they no longer are.
•  The yearly salary of automation engineers is significantly higher than manual testers.  My preference would be to have a smaller number of automation engineers focused on unit testing, web services testing (more expensive, I agree, than GUI automation engineers) with a few more manual testers. In simple terms I can replace every GUI automation engineer with two manual testers, a good trade-off for test coverage and for our economy with more workers working.
• From a career growth perspective, QA engineers have a better career path because those doing automation are working with languages like Java and C# as opposed to tool-specific languages (like SilkTest 4Test as an example).
• The cost for each business will vary from company to company.  However, analyze the bugs in your product for the last year and ask yourself:
  • What % of bugs would have been found with unit or web services testing?
  • What % of your application bugs were platform specific?  With those bugs would a manual tester have found the other UI bugs as well?  You need to invest in testing the other platforms anyway and you cannot do it with most functional automation tools.
  • How many hours per week do you spend in maintaining your automation?
  • When you run your entire automation suite, how many hours of manual testing are you replacing it with?
  • For the location that you are hiring at, do the skills exist to hire/replace existing people will needed?

There are clearly more questions but those listed above are the most critical.”
Also, it is not so much the extent to which the testing is automated that matters. Dorothy Graham says it could make sense even to automate just 2% of your testing, and I would agree. The question is whether the investment pays off, not how much you invest or how much you automate. An for a pilot project, trying out automation on 2% could also be a good idea.

I think what determines ROI is (mainly) the following:
- The setup cost (licenses, perhaps servers, training),
- the cost of creating the testware (automated test + supporting framework / software),
- the cost of maintaining the testware, and
- the cost reduction in manual testing.

John’s response:

“
You are right, of course, regarding the fact that you do not need to automate 100% of the test cases.  In fact, you would fail if you tried to automate all test cases, but that’s another discussion entirely.”
The setup cost depends on whether you buy or build, whether you use open source or commercial tools, etc. More and more people are finding out that free software can offer a lot of bang for no bucks (with no vendor lock in …).

The time and effort spent on creating the testware is interesting, but usually much less than the time and effort spent on maintaining it, as with all software. So let us consider the total effort. It depends to a large extent on:
- The development process (e.g.: how much time between releases),
- The product planning (e.g.: how much change per release),
- The test automation approach (an advanced one using a DSTL requires much less maintenance than Record & Playback; is automation complete when the product is ready for testing or is that when automation can start – DSTL=Domain Specific Test Language), and
- The skill of the automator (the quality of the DSTL, the maintenance sensitivity and maintainability of the testware).

The cost reduction I will leave for you to determine.
The amount of time between releases does not have to be an issue if the amount of manual maintenance is low enough. A good DSTL (high level, no irrelevant execution details, tooling details and interfacing details in the test) helps a lot by reducing the maintenance to the test. A good automator helps a lot by reducing the maintenance to the rest of the testware. The page object concept, for example, can be a great help here, both for web and GUI apps.

John’s response:

“
Again, good points.  However, the number of great automators vs. average automators ensures you never achieve the ideal situation above.  Even if you do, however, investing in other forms of automation combined with manual testing against the web front end is almost always cheaper.
“
One guy’s research indicated that about two out of every three test automation projects fails sooner or later. A major factor is that many are not aware that test automation is software engineering, and people with the wrong skill set do it. As a result, they may work hard, but not very smart. It is a bit like the guy who is asked what he would do to empty a bath tub when given a tea spoon, a regular spoon and a really big spoon. When he choses the big spoon to be able to work faster, he is asked why he does not simply pull the plug …

Of course, there are limits even to what a skilled automator can do. But in my experience of 10+ years of creating solutions for automated testing of all sorts of systems, maintenance was never a problem.
John’s response:

“
While I don’t know who this ‘one guy’ is, the numbers make sense.  Software projects fail at a high rate, a rate that is sometimes overblown due to the lack of a solid definition of project success versus project failure.  Automation development, if you ignore simple record/playback, is software development.

I am fine with agreeing to disagree but wanted to thank you for the detailed and passionate response.  Passion is what makes life great, don’t lose it.
“

Functional UI automation for web applications does not make sense

More heresy, I know.  However, after spending nearly two decades building and testing software I know that it is the undeniable truth. Please understand the following:

• I am only discussing web applications. My logic will make more sense in a few seconds.
• All other forms of automation both make sense, and are critical, for web applications.  This includes unit testing, scalability testing, performance testing.

My reason for being against functional ui automation is simple, the ROI IS NOT there.  Let me tell you a couple of stories that I have seen, and heard, in recent years:

• In one case a company had one automation engineer.  Due to the rapid pace of code change associated with iterative development the engineer spent as much time updating the automation as they did running and analyzing results.  Since frequent builds were being run the automation was often not executed for a few days as the engineer fixed it and analyzed results.  Bugs found were sometimes no longer in the code or, worse yet, had already been found by manual testers.
• In another case I have saw a company invest in a larger automation team, with one group focused on new development and one team focused on maintenance.  The total coverage of the product would have, in my opinion, been improved if they had simply switched to a higher number of manual testers with a smaller number of automation engineers focused on performance and scalability.

The argument is always made that the automated UI tests you build will save you time throughout the testing cycle.  It does not do so with web applications.

The argument is always made that these tests will cover the “boring” areas that people will not run through on their own.  Again, I disagree.  Important bugs in the UI will be noted by the manual testers.  Bugs in subtle areas (such as your security model) should be caught by unit tests.

Am I right?  Have I missed anything that would make this argument invalid?  Let me know what you think.  You can find additional commentary here.

John

Why is it that the best bugs are found at the end of a release?

It never fails.  I have worked with QA groups ranging from 1 to 100 and you always end up finding the most interesting, and sometimes the most critical, issues right at the end of a product release.   There are a number of reasons for this phenomenon, here are the ones I have seen most often:

• QA does not receive working features until the end of the release.  Release dates are fixed and everything comes together at the last minute.  We have all seen this and it always results in two things:  A mad scramble by all involved; A patch soon after the initial release.
• QA is properly staffed and works on “proper procedures”, developing robust test plans for each feature.
  • This is great, but sometimes leads to the basic aspects of features being tested with limited to no real world testing taking place.
  • Your QA team should be building test scenarios from the original business requirements and testing these as soon as they can.  If the customer can actually accomplish the the business tasks you were trying to automate they will be more forgiving of typos in a dialog.  However, they will not care that the product is free of typos if the core requirements have not been met.
  • You must build in time for random or freeform testing throughout the cycle.  Let your team be creative.
• Lack of customer testing.   QA teams do great but they are not your final customers.  Supplement developer and QA testing with customer beta testing early in the process.
• Unfortunately, many times, you also fail to test for performance, scalability, and security issues until near the end.  Ensure this is done throughout the cycle or you will have even larger issues to deal with late, when you least want to make these detailed changes.

John

Sunday coding is productive coding

Okay, truth be told I would have much prefered spending time with my wife and kids, but there are days where work comes first, today was one of those days.    At the end of the day though I’m happy to say we are looking good to posting a new version of the Swimfish Milestone Tracking Matrix tomorrow night around 10 PM ET.

While I’m mentally drained I did want to mention a couple of topics that are worthy of discussion, would love to hear your thoughts (either as comments or tweets), when you have time.

• What is the best way of generating a web-based installer for your web applications?  While I generate web deployment projects I have to believe there is a better way.  Has anyone had success with other methods?
• Why is it that the best software defects are found in the last 48 hours of a product release? 
• At the end of the release what processes do you follow to deploy new builds to your QA team?  We are constantly deploying new builds as fixes are rolled out.  What’s your approach?

I’ll fully explore these topics in upcoming weeks, looking forward to hearing your thoughts in the meantime.

John

Are you ready to release the product?

Checkpoints at key milestones are always important, but never more so than at the end of a project release. The Swimfish team is nearing completion of our Milestone Tracking Matrix product.  The product, which enables users to coordinate their deals efficiently and completely, from due diligence to distribution tracking, to discovering new opportunities, has a complex back-end that is tightly integrated with underlying CRM systems.

For this review meeting it’s critical to have the business owners (Product Manager in our case) as well as the leads for support, development, QA, and documentation in the room.  Here is a summary of some of our checklist items:

• Make sure that everyone is on the same page in terms of what new functionality is being included in the release.  You should not be surprising anyone at this point.
• Review the status of any remaining open bugs.  Do you have any issues remaining?  What is the risk level with fixing, or not fixing, the remaining items?
• Ensure we all understand the status of key pieces of documentation.  In our case this includes release notes, end-user help, and server administration guides.  Are they completed?  Who has reviewed them?
• What is our customer notification plan?  Our software runs in our SAAS environment and on-premise.  Ensure your plan delivers the right message to each audience.

These were some of the key points we covered today.  How does this compare to your end of release meetings?

John
On Twitter at JohnFMoore

I didn’t have time to follow that process

As I sat with my team for our post-mortem I heard someone state that they didn’t follow the agreed upon process because things became too hectic.  Now, this person is great.  They are one of the hardest working people I know and they are a critical part of the team.  However, this answer is never acceptable, never.

Processes are put in place to stream-line our jobs while ensuring that the job is done right.  However, if you have people that are not following the established processes, you need to find out why.  Here are some things you should be asking:

• What reasons are people giving you for not following the process?
  • Do not jump to conclusions.  Have a conversation and listen to the concerns people are raising.
• Is it one person avoiding the process or multiple?
  • If it’s one person determine if this is a poor performing employee or just someone stuck with a poor performing process.
• Is the process still valuable? 
  • Sometimes processes remain simply because they have always been there.  Just as Tevye had to ultimately turn his back on many traditions that were outdated (Fiddler on the Roof), you may be in the same position.
• Do people understand the process?
  • Good processes that are not well understood quickly become bad processes.  Do you regularly train new hires on your processes?  If necessary, do you regularly run refresher courses for your teams?
• Is the process optimized for the world you live in now?
  • If the process is valuable, and it is well understood, it may just need updating for the world we live in today.  When I worked at Lotus in the early 90s we had staff on hand to type in hand-written bug reports.  Bug tracking was a good thing, having someone spend hours entering hand-written bug reports would not be valuable, however.  Ensure your processes take advantage of modern tools and behaviors.

John

Post-mortems in startups

Post-mortems can often be a large waste of time.  I remember years ago setting in large conference rooms with a hundred other people, grouped by tables of 8 to 10.    We would spend anywhere from a half day to a full day discussing problems we encountered on the project, possible solutions, all the while filling up reams of paper with ideas.

It was exciting at the time.  Dozens of bright people thinking about all the great things we could do better, and committing to solving all of the project woes the next time.  Next time we would get it right.  The problem, of course, is that all of those reams of papers rarely translated into real change.  We would all go back to our daily routines, busy with our “real jobs”, and we would often repeat all of the same mistakes.

We were not the exception.  Many teams try to use post-mortems and set unrealistic expectations.  I’ve used the following simple approach for a while and it works; it can also work for you.  For those that work for me now, this is the post-mortem process that begins this week.  Here are the keys:

• Do not start the process too early in the startup’s existence.  In the beginning you are focused on getting product to market and are defining a baseline process.  You need to give it a few months to shake out.
  • I know this sounds backwards, why wouldn’t you focus on process improvement right at the beginning?  As your team is coming together you must have a strong hand on laying the initial framework.  You need the team’s buy-in but you must also steer the ship with confidence and poise, letting people’s creativity fully flow into getting their core jobs done.  Once everyone has hit their stride begin looking for improvements.
• Make sure everyone understands the following.  The processes that are currently in place are acceptable for where the company currently is, but not for where it must be in 3, 6, 12 months.  You must continue to get better or the company may not exist in 12 months.
• The post-mortem must be a regular part of your schedule.  The optimal schedule for me has always been 4 – 6 weeks.
• When you meet, here’s what you should do:
  • If you are the manager of the team DO NOT TAKE OVER THE MEETING.  Facilitate the meeting, make sure everyone is involved, but you are just one voice in the room.
  • Make sure everyone has a chance to speak about what challenges they encountered since the last meeting.  Don’t propose solutions yet, just note the challenges.
  • Ask everyone what worked great.  Make sure you applaud the things you are doing well.  I am certain you’re doing more right than wrong, don’t lose sight of this.
  • As a team, identify the one or two biggest pain points from the list you’ve come up with.  One is best.
  • Once identified, discuss how you will eliminate or at least minimize this one items impact between now and the next meeting.
  • Now, the important part.  As a team, commit to improving this one item.  A single item does not sound ambitious.  However, in a startup the insanity of your daily lives makes it hard to commit to solving too many problems.  Pick the biggest problem and eliminate it.
• When you leave the meeting, summarize the meeting, noting what has been committed, and schedule the next meeting.
• Now, the important part.  With every daily standup, remind people of the post-mortem committment.  Make sure that everyone is following through on the changes the team agreed upon. By the next meeting it will be a natural part of your process, ensuring you have one fewer painpoint.

Do you use post-mortems today?  Do they work or are they inneffective?  Let me know.

John

Are you eating your own dog food? If not, no one else will…

Listen.  Are you developing products that people in your company could be using?  If you answered yes and they are not using these products you have a big problem.  If your teammates don’t want to use the products your customers, and potential customers, won’t either.

Early in my career I had the good fortune to work at Lotus.   I worked on Lotus 1-2-3 and it was a great time in the industry and a fun time to be at the company.  We were producing a leading product that solved real business problems.  In addition, the company was full of bright, passionate people.

When we chose to follow IBM and build on OS/2 we found ourselves quickly behind this young upstart called Microsoft who was creating this operating system called Windows.  Who knew that it would take off, who would want anything but DOS?  Well, Microsoft came out with the first real spreadsheet product for Windows, you might have heard of it, it’s called Excel.  Well, even though no one wanted to admit it at the time, people at Lotus started using Excel instead of 1-2-3 to get their job done.  People were not willing to use the free (internally) tool that we were creating.  The battle was lost.

Back to my original question.  Are you developing products that people in your company could be using?   If they are not using your products change this immediately.  When I joined Swimfish I found that we were using Lotus Notes for everything.  We host sharepoint as a SAAS offering for customers and were only lightly using SharePoint or the integrated CRM product that we created.  It was time for a change.

• I drove the company to commit to SharePoint as a platform.  We still use Lotus Notes but are now using SharePoint whenever possible.  If you have any doubts, check out my SharePoint in Action posts in this blog.
• People had some early frustration with the shift to SharePoint.  While they were excited to leverage the product it was new and it was lacking required functionality.  Today we have a much better offering, people really understand the platform, and our customers are getting a better solution.

Engage your teammates to understand what they need in the product.  Find out:

• What problems are they unable to use the product to solve?
• How well does the product perform?  Are they constantly encountering bugs?  Is it too slow?
• Ask them, if they could use a competitor’s product to get the same job done, which one would they choose and why?
• If they do encounter problems, how easily can they get answers to the problems?  Is the documentation helpful?  Is the support team knowledgable?

Learn from these questions and make your product great.  If they’re not eating your dogfood they’ll be eating someone elses.

John

Why I love the daily standup

I am not an agile evangelist.  I am not a waterfall evangelist either.  My favorite engineering processes are entirely driven by the need to successfully overcome product development painpoints and ensure that we build the best products possible with the least amount of friction for everyone.

I also do not believe in status reports for individual contributors.  Line managers need to be very involved with their teams and know the details of what their teams are working on. They need to know the risks and help mitigate those risks. They need to be a buffer for the team to ensure that the team is able to deliver on their committments without interruptions from outside forces. This is why I love the daily standup.

Every day the engineering teams meet for 5 – 10 minutes and review the following information:

• What are they currently working on?
• What are the current challenges that they are working to overcome.

That’s it.  I want to know if you’re on or off schedule and ensure that everyone is on the same page about what their teammates are working on.

I also use the standup to remind people of our goals.  Everyone is busy, working on multiple projects, and it’s easy to forget the key priorities.  I like to ensure that everyone is reminded of the priorities daily.  Avoid surprises with constant communication.

What practices do you use to help kep your teams on the same page?

John