How can so many travel meta-search engines exist?

Seems like there are dozens (or hundreds) of travel meta-search engines like fly.com, Kayak, TravelPost, etc.

I can't imagine any of the "big" sites like Expedia, Travelocity, etc allowing their content or searches to be indexed and stored like that. Yet, it happens.

Do most of these travel sites have deals with these companies? What are the rules about that? I know about following the robots.txt file but I would guess many of the big sites have theirs so that you can't search their site.

Thanks for any clues.

They exist because travel (tourist) industry is huge. AFAIK there are 2-3 data providers for fly ticket prices in US (at least that one friend working in such company told me). There is standard protocol for communication with such data providers. You pay some subscription fee and probably arrange some standard deal.

The problem is that this information is changing each minute. Some web sites have exclusive data arranged by contract that gives them some advantage in providing lower prices. General problem here is to make optimization on uncertain data and to predict prices based on many factors and data sources.

The provider sites probably have a standard deal with search engines to give them a percentage for business. They don't really care who links into them, as long as they get the sales.

It's also pretty cheap to run an engine that accesses the various sites, so low barrier to entry and you get this result of lots of people trying. Market forces will take effect and simply reduce the margins down and a lot of them will drop out.

That said, if you run a simple engine using low cost labour, you can probably survive this and just about make a living.