Ironically Cobol programming is such a niche. Lots of big companies (like banks) are stuck with old Cobol programs that are so expensive to replace. In this niche, you cannot get in by simply offering a fancy new solution, you need to work with what they have, and maybe help them migrate to cheaper solutions. I believe this field is extemely lucrative as some banks are really in a desperate situation.
Also, the mainframe world in general is hard to enter, I believe as the equipment is expensive.
Also anything related to expensive software (like the software from IBM) is quite hard to reach. Say for example that you developed something for Informix.
Aviation/Transport, Medical and Defense software have high barriers to entry - perhaps not technologically, but because of the regulation and process involved.
hav
I would say
Prediction software. Like finding weak signals that indicate a trend before it's a trend.
Another area which is a darling of mine is what I call The Ghost Protocol http://000fff.org/the-ghost-protocol-digital-identity-for-immortals/ which is basically about building mirror versions of yourself, logging everything we do and create a digital "ghost" so to speak. This could be used in most forms of identificiation, reputation and automation.
High-barrier isn't the same as niche.
Doing business with the Federal/State/Local/K-12 government can be painful for the clueless. But if you understand what the target government's needs and priorities are, and understand the procurement process, that can work in your favor too.
Many companies disqualify themselves from a huge market or limit themselves to niche status because they are too lazy to do some paperwork or partner with other companies..