This awareness I saw it growing at Enterprise Search Customers/prospects from early 2000's on. I was part of Vivisimo (spin off from CMU) which created a "clustering engine". In the beginning we always had the "Librarians", pro categorization & classification, dismissing the interest of Clustering (or maybe they were justifying their job). Then more people in the organizations (government or corp.) said yes those 2 approaches (automatic "on the fly" clustering vs human generated taxonomies) are complimentary. Finally couple years ago, in the same week, I visited the Government of Israel which said "we just need clustering" eg. on the left side: http://search1.gov.il/govilt?query=israel and then Ferrari competition in Maranello (yeah:) even if I don't like cars;) and there their CTO explained to my surprise [was trying to tell him "we integrate well with your existing taxonomies, ontologies..."] "forget about any hierarchy or classification tree in our File Systems, we just search!". Reality/information is much more diverse & evolving than any predefined categories.
You can combine the two approaches, with "just search" which also uses human "categorisation and classification". This helps an awful lot with relevancy ranking.
In smaller data sets (eg not Google scale), where all the items are about roughly the same thing, you need some human classification for your search to work well.
Yes you are right. I was a big Friendfeed fan and proposed ideas which we implemented: humans could modify the search index on the fly with ratings, comments and tagging of the search results. Eg. people were able to search documents based on people's comment or rating for instance.
I think the lesson from "web 2.0" apps is that if you make something easy to use and useful, people will use it. So the rise of folksonomies is about usability, not doing away with structure.
In my experience, if you can make entering more structured metadata just as easy, people will enter it, and you get a big return in the ability to use the information you've collected.
