I'm getting ahead of myself here because my product hasn't launched yet, but I thought about this and would appreciate the thoughts of this community: Imagine this:
You're launching a great app (or any product). Your research indicated that it fills a need, albeit a niche. Now, having observed the sales for a reasonable time, you see that the sales don't really take off; in fact, after the initial flurry of friends' purchases, the sales don't do anything much. You're understandably disappointed. How do you decide between putting more money into the product (advertising etc.) versus calling it quits?
Entrepreneurship is about solving TWO problems: value creation and value distribution. If you've initially "marketed" to your close friends, then you've never gotten real customers and so you never really distributed your product. That's ok. But let's go straight to the meat: do you have a viable product for which there's a demand?? The best way to tell is by simply observing what your friends are doing with it. Are they using the product? How often? What do they do with it? Are they talking to you about it? In other words, if you have a product that creates value, then DON'T QUIT. If on the other hand, people aren't using it, or usage is declining fast after an initial curiosity, then you haven't created something of value; quit.
If you have a product that creates value but you're evidently not able to distribute it yourself (and that's totally fine), then I'd recommend you find a person to manage the distribution instead of you continuing with business development.
So the answer to your situation will come by looking at reality and answering this simple question "Do I have something that creates value?" Good luck and let us know.
Easy!
When your heart's not in it anymore, call it quits.
There's a really interesting book called The Lean Startup by Eric Ries, which discusses this point in detail. He calls it the "Pivot versus Persevere " decision. The book lays out a framework for making a data-driven decision.
The premise of the book is that the linear model of product development - spending a lot of time planning, designing, implementing a product, without releasing a product to test market acceptance and product-market fix - leads to wasted time and money. This can be avoided by creating testable hypotheses and incrementally releasing products and/or features which test these hypotheses. The first releases might be minimal viable products which, while not complete, offer a way to test the initial assumptions ("leaps of faith") and enough value to interest customers.