As competition in the search space continue to heat up, I think the space is about to enter a critical phase in its evolution in which walls and roads will be built around and into popular or valuable areas of the search space.
The idea is not to restrict access but to restrict discovery or limit quality of search results, meaning that, while anyone can read those pages, they will have to use a specific search engine to find those pages or to get quality search results.
Walls will be erected around popular areas of the Web so that those areas can be searched only through a particular search service. Major areas will be bought, shared, or locked in via some affiliate restrictions.
MSDN could be searchable only using MSN Search. Likewise, Blogger.com blogs could be searched only through Google. Amazon website could be made searchable only through A9. Throw in alliances, affiliates (Amazon), and service dependency (AdSense) and time into the picture and the result is a the segregated Web.
How this phase unfolds doesn't have to be as ugly as I described above. More palatable scenario is to open up previously unsearchable areas of the Web and make it available only through a particular search engine. Another approach is to enrich search result using metadata not available to crawlers.
I am neither worried nor find the notion of walls in search space chilling as John Battelle wrote in reference to this post. All businesses are, in essence, selling access to goods and services. For goods that cost little or nothing to reproduce, selling access or quality of access is a legitimite business model.
Today's search engine results only show what they can scrape from the surface of the Web. In order for them to provide higher quality search results, they have to dig deeper, into areas not directly accessible from the Web. Whether or not to sell/leverage access to those areas is a business decision, not a moral one, unless there are privacy or ownership issues involved.