Deciding how broad or narrow a market to aim for

When you're not sure which market is worth aiming for, how do you figure out whether to go for a broad market (with lots of competition) or a narrow one (with no competition but good chance to slip by the wayside)?

Here's the situation:

I've got this idea for a software startup. At the time, it was a niche idea that could be an educational aid, though I was thinking on an individual basis. I evaluated the competition at the time for the niche idea and didn't see any viable competition, but I worried about not being able to attract customers.

I was given a suggestion to try to market to institutions (i.e., high schools and colleges) and go for a software-as-a-service approach instead, I guess with the idea that a more reliable revenue stream could be formed with annual subscriptions from institutional accounts. Running with this, I figured that I needed to expand the focus of my startup initiative, so that there would be more features that could provide broader appeal and justify my charging an annual subscription fee. I was writing some code that started to cater to this approach, but I realized today that I needed to reevaluate my competition. I was entirely unsurprised to find that going with the broader focus, I suddenly have lots of potential competition, some of it being well-established.

I'm in a state of barely-controlled panic now, trying to figure out if I need to restructure my idea yet again, and of course wondering if maybe there isn't a way I can enter any relevant market at all with my idea. I'm also realizing that I need to restructure my approach to do a little more research, so I don't waste too much time going down a path that won't prove successful in the end.

What sort of logic should I employ in figuring out what features and markets I should be thinking about in the pre-startup stage?

I'd have to see your narrow and broad products and research the competition before I felt certain of my answer, but I am pretty confident:

Stay small, focused and code-light. Establish customers, revenue and most importantly, profit. Once established, Rob's ideas of expansion through partnerships is a great one.

You do not want to compete with well established companies on technology. Its a hard battle to win and it never ends. You've discovered a potentially untapped niche. Go with it.

Most importantly:

Do it. Don't plan forever. The company that takes the wrong action quickly will reach the right action through correction far quicker than the company who debates which action to take for a long time.

Once you're IN the business, taking orders, talking to customers and managing your live product, you will KNOW the right answer to the question.

Yeah we hit this issue a lot as well ... do one thing well or hit a much larger potential market ...

Most of the time we pick one thing well ... establish it, then move onto the thing next to it. This gets your brand established away from competition, then you can move out. We also make our systems open so we can integrate to up and down stream "almost competition" systems. This has the benefit that we are an easier decision for prospective clients who are evaluating our system "oh, it already works with what we have" and makes us tempting buyout targets by larger players as we are playing in the space and can extend their product offering.

So, in summary, if you have enough of a market available to you in the focused version, stick with it until you have traction, then consider expanding.

The other consideration from the development perspective is that as you start competing head on there is a LOT more code you have to write to keep up and stay ahead. Focused, you development footprint is far less.

... Lastly, if you have a complementary product you can, carefully, approach other established vendors in the space and offer them resale of your product (much easier than trying to do it yourself).