What business models exist for mobile applications beyond the AppStores that are run by the platform owner (Apple, Google)?
The standard business model seems to be low priced apps in 3 segments (<1$, <10$, >10$) with a large potential customer base, where the AppStores provide market outreach, scalability and little upfront investment in a sales process. AppStores are perfect for this kind of market.
Applications that are content-driven need a recurring monetization, so they remain profitable for a publisher. AppStores fill that role too.
However, there are cases where the above business models aren't reasonable. This is especially true in a niche market, where the number of potential users is low compared to the development cost but the achievable price is high nonetheless. The benefit AppStores provide in this segment are low in relation to their high cost (typically 30% of the revenue).
What alternative business models work in this segment? There are applications (such as medical imaging) where an alternative business model has likely proven itself? What are those? Can they even be legally implemented, with respect to the usual restrictions put on mobile platforms?
Here are a couple of alternatives that seem to be used (just for thought-food):
Sales Business Model Mobile Apps
If it's an app that is very expensive to buy (e.g. $1,000+ per user), then probably requiring a specific device to run it might not be a problem (e.g. Android with non-market applications support enabled, which is very easy). It really depends on the niche though.
If it's something less expensive, as you noted, a good way to sell licenses would be a free app on official stores that are actually tied to a paid online service, and you would not even need to work with official enterprise app distribution channels.
I would completely avoid unofficial stores as they are simply unknown to most users.