How to do marketing research in Finance related software?

I am a software developer in Canada. 2 years ago, a close friend referred me a business owner who is running Futures trading firm(12+ employees) in Chicago. He need some system to help them manage trading strategies and formulas.

Friend asked if I want to join because I was working in a financial firm and know a special technology to solve the problem.(Using expensive IBM servers) I did not join the team. They hired few full time developers to build the system for 2 years and could not solve the problem.

I realized that this might have potential market (targeting small trading firms). And now I am ready to build open source system to solve that type problems. But I want do some marketing research to make sure there really have buyers there. Because most of the firm is hiring developers to build custom system in house, I am not sure if my solution will be a good fit.

Try to contact the business owner, but he left the company. Is there anyway to get in touch with those trading firms? Is it okay to make cold call to ask if they need the product? Any better idea?

I would recommend you checkout http://quant.stackexchange.com/ to see if there are any questions about your particular area of expertise. If so, perhaps you can get a handle on the level of interest, number of people discussing it etc. Further you may be able to ascertain the kinds of problems/issues they are having as well as the level of sophistication.

Once you have that kind of understanding of the current state of the art you will be in a much better position to know the potential market as well as the level of effort required to develop a minimal viable product.

As far as cold calling; sure it is OK but probably not particularly productive. It would be much better to identify the key players by name through places like quant.stackexchange and either private message or email them. People are generally much more receptive to dealing with people that they at least have some familiarity with through a forum than with someone they have never heard of.

The situation you are in will be somewhat problematic. Here are is why.

You could make a cold call to a larger hedge funds and eventually find someone who will be interested in looking at your model but here is the rub. By what you are describing you are doing statistical modelling for portfolio optimization or trade execution, much more likely the former. In that field companies are looking for a competitive advantage so if all of them have a common model what is theirs? Here is another reason. The model training that you will have to do requires data from exchanges which is quite expensive to get. Thirdly the new trading firms don't like using Big Iron. It's too expensive. Using commodity hardware running Linux or Solaris on Intel and the like is the way to go running on Matlab, S, R or the like.

So looking at your options the best solution would be to start trading and find a firm that would allow you to run your own model trading some of your money and some of theirs.