How long of a closed-beta is enough?


3

We have developed a product offered as SaaS, that helps businesses to manage their entire business online (details at www.trakeze.com). We have been running a nice long (more than 6 months) closed-beta. We have received very good feedback that keeps coming and we keep incorporating it into the product, and we think we are ready for the production/real customers. I know people say, you should get the first version out quickly, so that customers can start to look at your product, use it and help you improve, but until this time we have been working on making the product better and better. Question: Is that beta duration good enough, and should we start with free trials - otherwise, I feel like, there will be no end to the feedback. (Most of the feedback now is around layout or ways to use the features and not necessarily the functionality)

Software Saas Beta

asked Jan 24 '10 at 04:45
Blank
Puneet Gangal
281 points

5 Answers


3

If the beta testers have been using the product for 6+ months they will not notice the problems a novice will face anymore (the problems someone using a trial will have). You need to bring more people in (have the beta testers invite their friends a la GMail) or open the product to the public.

Which way to go only you can decide, but since you ask the question you are probably ready to open it to the public.

answered Jan 24 '10 at 06:21
Blank
Peter Olsson
400 points
  • It's not because the product is in private beta since 6 months that all beta testers are using it since 6 months ;) I'm in "private beta" since nearly 4 months and I'm sending about two invitations per day to new "private beta-testers"! – Tristan St. 15 years ago
  • Agree. We have found that in our beta. We try new users hoping to understand from their perspective what "might" be confusing to a new customer. From that feedback, we have added a lot of in-line help. You are right - we are finding usability type things now - the feature set if all complete - which is why I think we are ready. – Puneet Gangal 15 years ago

3

Key questions - is the quality there, major bugs ironed out? is it competitive with other products you'll be competing against? does it have the core set of features to deliver the solution it's intended to address? What's the worst result if you do come out of beta and launch? What do you fear?

It sounds like the product has the core functionality it needs based on your current beta testers feedback. Even if you aren't superior to competitive products, you can get there. Or maybe you are already. To me only the first point about quality could be a deal killer. That would be the only reason to stay in beta.

Get it out the door. It seems like all signs are point towards that. You can keep improving the product, addressing competitive issues and so on once you're launched.

Best of luck,

answered Jan 24 '10 at 07:58
Blank
Chris
4,214 points
  • Hi Chris - Thanks for the response. The quality is there. We have the core set of features well developed, tested, beta tested, ready to go. We are excited, ready to go - but I guess, the fear is once you launch, you only have one chance with each customer. – Puneet Gangal 15 years ago

3

Beta test is usually conducted on feature frozen product and main goal is finding bugs.
You should draw the line and release the product. Otherwise, you can keep beta testing and adding new features for years.

Another reason to release ASAP is that your product is SaaS, so you can release new features and (more important) fixes to all customers as soon as they are ready.

Edit: To answer Jeffs comment:

I think that you can go with much shorter beta test period with SaaS just because you can immediately release updates to all your customers.

You need to be sure that your first release is usable (not crashing). But your first release does not need to be feature complete, it just needs to have enough features to be usable. Look at Puneet's site. He has suite of at least 6 different web applications and it looks that each of them can work standalone. He could beta test and release them one by one. Beta testers would have smaller applications to test, so response would probably be of better quality.

My experience with web application is something like that. We released HR SaaS application aimed at local government. First version had smallest feature set our first two customers expected. Our beta test lasted for about three weeks. That was the time we needed to fix issues our beta testers found.

Further releases were relatively small increments to initial release. We were also ready to revert application to previous version if something goes really wrong. We never again had beta test since each new feature release was small enough that we were confident there is no critical bugs.

While we had desktop application, we did less frequent releases and each release had much more new features. This resulted in longer testing periods where we had beta testers.

answered Jan 24 '10 at 14:02
Blank
Zendar
171 points
  • Do you see SaaS models needing a different approach to Beta if releasing new features and fixes is actually a feature in itself? – Jeff O 15 years ago
  • Agree about time-to-release aspect w/ SaaS. One clarification: even though there are 6 applications that help w/different business functions, they are all connected (just like in the real world!). You can't manage project tasks without having projects, which stem from sales opportunities. You get the point. This "connected" aspect alone makes us unique (otherwise there are 100s of offerings that provide great CRM, HR, timesheet). Our beta has been longer because of this, & we have been very diligent, as we want to ensure that information flows in a manner that it doesnt confuse the customers. – Puneet Gangal 15 years ago

2

Are you ready to handle the volume of more customers? Once the beta is over, the user expectations for speed/performance is going to increase; paying for things does that to people. When it is a beta, people may over-look little bugs and interface inefficiencies thinking you will fix them before release (How could you not know these things exist mentality.). Make sure your support, purchase process, billing, and disaster recovery have all been tested as well as the software. It would be a shame to have a great and well test product only to lose customers because they don't have faith in the company behind it.

answered Jan 24 '10 at 13:28
Blank
Jeff O
6,169 points

1

We used to say:

Beta is like Fight Club. It goes on as long as it has to.

Meaning: We want to get rid of the obvious bugs and confusing points before we say it's done, rather than draw a line at a certain amount of time or other arbitrary measure.

I think you have a good feel for when the influx of requests starts to abate -- when the volume lessens and/or when the severity of the problems lessen.

Another thing to watch for is focus. When everyone is complaining about the same thing, fix it. When everyone's complaining about a different little thing, you're fine -- that's normal product equilibrium.

Some of our betas went 7 months; some 1 month. That's OK!

answered Jan 26 '10 at 02:18
Blank
Jason
16,231 points

Your Answer

  • Bold
  • Italic
  • • Bullets
  • 1. Numbers
  • Quote
Not the answer you're looking for? Ask your own question or browse other questions in these topics:

Software Saas Beta