How can I judge the reliability of a software product?

I am considering buying some software, which would do very advanced calculations/simulations for me. If this software provides wrong results, I could go bankrupt.

I specifically did not want to mention what kind of software I am talking about, as I am interested in a general answer.

My question How can I judge the reliability of commercial software?

My best ideas so far are:

Version number (horrible)
Brand name (horrible)
Large or small company developing the software (does not say much either)
References (it does not say how much the large company depends upon the software creating correct results)

What other factors could I use?

EDIT: A couple of you have mentioned testing, which is fine, but in my case it could take months of work to do more than basic testing. I have considered hiring someone do to this, but it seems like I am really doing the job of the software company. In addition, would only be able to do black box testing.

Purchasing Sub Contractors

asked May 16 '11 at 22:19

David
1,567 points

@Tim The existence of my startup (and others in a similar situation) will depend on the decision, so it should be on topic. – David 13 years ago
@Tim this is a valid question since many small businesses lack the resources to do a full analysis of software options, and need to know the best way to evaluate a purchase for quality and ability to meet their needs. – Elie 13 years ago
Well, small businesses also need to evaluate cars, copy machines, etc. Large businesses also need to evaluate software - as do individuals. It is not specific to startups at all. It is specific to this person's needs, and he does not give enough details to let anyone help. If the software is that important and could bankrupt the company then it is very specialized. This is not a good quality question IMO. – Tim J 13 years ago
Thank you @Tim for share with us (especially @David) your reasons for flagging. this type of immediate feedback is really helpful. I have opened a question on Meta to discuss the issue that you raise. That is the appropriate location for the continued conversation. http://meta.answers.onstartups.com/q/483/8791 – Joseph Barisonzi 13 years ago
@Tim I believe in asking question in a so general way that they are useful to others. Your statement: "If the software is that important and could bankrupt the company then it is very specialized" is just plain wrong. There are lots of companies, including startups that rely heavily on software. How many startups would not go bankrupt if Microsoft suddenly released an update to Excel, which calculated wrong? EDIT: I just saw the comment about the thread on meta, and I will continue discussing there. – David 13 years ago
I'll vote to reopen if it does get closed. The question is fine. I still think it is not startup related - other than a startup has this problem, but it is actually far better than a lot of other questions. I have changed my mind – Tim J 13 years ago

4 Answers

David: I've been in the software industry for over a decade, and to be be blunt, I don't see how any of your ideas could give you an insight into the reliability of the software you are considering.

With one exception: References When the software is used for highly complex or mission-critical tasks, your best bet is to look at who the existing customers are. You seem wary of this because a company could use the software only rarely/have low operational dependence on the software. I would counter this argument by saying that in a high-risk environment companies will not ever use - in any part of their process - software which is not reliable. They can't afford to.

So, look to the case studies, search out discussion boards and blogs (they will be out there), look at the list of users the company has on their website.

And once you make the decision, do a test drive - this is easier/more prevalent with web-based/thin clients than it is with thick clients, but speak to a rep and see if you can do a test drive, and check the results for yourself.

Best of luck.

answered May 16 '11 at 22:30

Gef05
278 points

References are provided by the company for the clients with which the software worked -- has anyone ever been successful in "Could I have some references from people who used the software with disasterous results?" – Joseph Barisonzi 13 years ago
As someone who has been involved in numerous startups and small businesses, I can assure you I would never pass on info from an unhappy customer. Period. – Gef05 13 years ago
Although you won't hear about the bad experiences, you can still tell quite a bit about a product from a happy customer. Yes, you won't get a product bashing out of it, but even happy customers may have had some smaller negative experiences that they will share (e.g. it's great, though we had some issues getting the reports printed properly, but the company helped us resolve that). – Elie 13 years ago
I agree. References are an important part of the qualification process. Is this product real? Qualifying the references of course is also helpful. But it can't be the end all. If it is mission critical software -- a recommendation will not validate if the software will meet your business requirements. – Joseph Barisonzi 13 years ago

After you have qualified the software as doing on paper what you need:

I would propose you have to: Test it. Develop a test of standardize calculations that you will be running. Run them on the software. See with what reliability the correct answer is delivered.

If they wont give you a version to run test on -- then it is not the software for you.

answered May 17 '11 at 00:33

Joseph Barisonzi
12,141 points

A couple of you have mentioned testing, which is fine, but in my case it could take months of work to do more than basic testing. I have considered hiring someone do to this, but it seems like I am really doing the job of the software company, I am would only do black box testing.

No. Absolutely no!! The testing will enable you to establish if it's producing the figures/results you expect to see in the format in which you expect to see them. This speaks to business rules, selection of mathematical models, etc.

It has nothing to do with bugs.

answered May 17 '11 at 05:53

Gef05
278 points

But it would really be nice to save those months of tedious work and focus on the real value adding. But so far, it seems like there is no quick way around this. – David 13 years ago

-2

The company does not provide a trial version of their software? How about a money back guarantee?

If they do not provide either, I would get in touch with them and talk directly to the sales people, letting them know your position. They would either provide you with enough info to boost your confidence in the product or be lenient and have some kind of trial period for users to test the software.

AND

Buy through Paypal - buyers are always more protected than sellers, especially with intangible goods. If the software is not what they touted it to be, let Paypal know and get a reversal.

Go blackhat. Get the software bootlegged, probably an earlier version, test it out. If it is what you need and is a success, I highly advise buying the software, don't just skip out on the developers that made a great product (if it is great ;)

I would scour Google and try to find people that have been ripped off, but make sure you get quality feedback - not some guy that couldn't find the button that was in his face.

answered May 17 '11 at 01:57

Flignats
37 points

you can downgrade all you want, but the reality is that when you have a need for software to get you from point A to point B, it could make you go bankrupt, and the product doesnt have a trial period - blackhat is an option. – Flignats 13 years ago
Just because something is possible doesn't make it a viable option. And by viable I mean life-affirming. You are free to suggest it but others are free to note its cost. – Kenneth Vogt 13 years ago
It's not a suggestion, it is an option. Don't be too narrow. – Flignats 13 years ago
I appreciate that @Flignats has posted the option (suggestion -- I am not sure I can tell the difference) of going to Blackhat. It provides the community the opportunity to express it's perspective on that option. That is what the community board is about. As someone who works hard to help companies monetized their IP and develop software-based products and services I feel good that the community has responded so quickly. – Joseph Barisonzi 13 years ago
I really appreciate the portion of @Flignats answer which is to talk to sales -- I think that this is really important. And I strongly feel that a company that won't let you kick the tire a little for a mission critical software is probably not the right one for you to bet your company on. – Joseph Barisonzi 13 years ago