June 8, 2006
The Next Generation of Search
More and more I find myself thinking, “Those guys at Yahoo get it.”
The latest example is Andrei Broder from Yahoo! Research, who, at last month’s Future of Web Search Workshop, gave the keynote talk titled, From query based Information Retrieval to Context Driven Information Supply [link is a PDF].
While this is a CTO-level presentation (i.e., high level, few details), it was well illustrated and to the point (and quite funny in spots).
According to Broder, Classic Information Retrieval makes all the wrong assumptions for the web context: classic IR ignores context, ignores individuals, and ignores dynamism (which is to say, the corpus is static). This is one reason I’ve never put much faith in academic search criteria (such as the TREC corpii).
He goes on to outline the first three generations of web search: from keyword matching and word frequency analysis (1st generation); to link analysis, clickthrough feedback, and social linktext (2nd generation); and, currently, in the midst of “answering the need behind the query” (3rd generation), which is mostly about supplementing core search with tools (spell checking, shortcuts, dynamic result filtering, …), or with high-ranking, high-certainty results from verticals (maps, local searches, phonebooks, …).
And what of the newborn 4th generation? It’s about going “from information retrieval to information supply” (emphasis mine), which is all about implicit searches: personalization, recommendations, …and, of course, advertising.
If you know me, it’s this 4th (and possibly 5th, see my notes below) generation that I’m always harping about. I wrote a bit about the future of search last year. I make a bit of an ass about it sometimes, but it turns me on.
And advertising, of course, is the big payoff from a corporate POV.
His final slide (slide 49) lists the challenges with this 4th generation of search: it involves a lot more data collection, a lot more data modelling, a lot more math, and a lot more understanding of the significance of the relationships between users and content.
What I find even more interesting, though, is what Broder left out:
- Search is well on its way to being integrated into normal navigation (faceted search is just one step)
- Social networks can, and should, affect relevance (social search is just one step)
- Search is being used as a platform, and soon, partners will be able to affect relevance for their users–providing yet more information to the core search system
- Search will soon be but a mediator between users and content–whether integrated into normal navigation or not–which provides the missing context for advertising, which can not be merely gleaned from content matching, or third-party user profiling
Did he stop short because much of the future of search makes Yahoo’s current search business positioning irrelevant?
Or because he has his team secretly working on the 5th generation of search and doesn’t want to give away his edge?
Or maybe he just wants to underpromise and overdeliver…
(Like via Greg Linden, who is one of the few bloggers I’ve made time to read in the last two weeks.)