Domain Expertise: The Key to Next Generation Search

There has been so much talk about Web 2.0, Web 3.0, the Semantic Web, blah, blah, blah. I feel like I am back in my days as a derivatives pro - where the use of fancy words and domain-specific jargon created an aura of sophistication and complexity that made us seem really smart to our clients. This was, of course, a bunch of garbage. Can derivatives be complex? Sure. So can genetics, molecular biology, and and a billion other things. But their essence can be distilled in simple and elegant ways so as to make their principles accessible to all. As a non-technologist Wall Street guy, what does all of this fancy nextgen techno-babble really mean?

To me it means getting the most out of the data. Which requires a few basic things:

1. Getting the data
2. Putting the data in an analyzable format
3. Applying statistics and/or linguistics to extract meaning from the data
4. Presenting the data in an easy-to-use manner tailored to your audience
5. Providing tools for analyzing the data based upon client requirements

Ok, that's cool. But there is an important point that needs to be made here. Steps 1 and 2 can be done by really smart technologists with skills in harvesting and database architecture. And this is important. But Steps 3 through 5 require something entirely different, something besides raw technical brains and skill in order to deliver a high-value product to the client: domain expertise. Why? Because the algorithms used to analyze the data need to be optimized for your particular domain, be you in law, medicine, travel, or even finance. Systems embedding artificial intelligence and leveraging machine learning require context, and that context can only be provided by those who know what the hell they're talking about, people who really get what their clients care about and are looking for. And this is not easy. This is hard. If there is one thing I have learned during my relatively brief time in the world of vertical search is that the value stack is pretty complicated, but all roads lead to a common theme: it's all about the data. And it is absolutely critical to overlay domain expertise onto leading-edge technology to get the most from the data. Period.

Today we announced some really great additions to our team at Monitor110, people with incredibly deep skills at the intersection of bleeding-edge technology, fundamental and statistical investement methodologies. You can just imagine the dialogues among development, product management and applied research. We all share the common goal: help our clients make money. But everyone is coming at it from a different angle. And this is a terribly exciting thing to watch and be a part of. This game of next-generation search - by whatever moniker you may choose - is in its infancy. And powerful machines and programs simply aren't enough. It takes people - great people - to make it happen. It is going to be a fun ride. And it really isn't that hard to explain when you cut right through it.

I can't tell you how excited I am for 2007. It is going to be a great year for technology companies everywhere. Talk about the Great Leap Forward - that is my name for 2007. And Monitor110 is sure to be leaping right along.

Comments

[NitinK]

Roger:

Excellent post! But par for the course for this blog - which is why I'm a regular subscriber.

I agree with you 100%. I think one of the key advantages vertical search engines bring to the table over the large general-purpose engines, is domain expertise. This allows them to provide so much more within their chosen domain: specialized features, additional datasets, related industry services, unique UI paradigms, parametric search and so on.

Recently, when I asked several leaders in the Vertical Search space for their reactions to Google's new vertical initiatives - OneBox, Co-op, CSE - their responses were surprisingly consistent given their widely divergent domains: the VSEs differentiate themselves with a strong focus on a particular user set and on solving domain-specific problems for that user set. More details on their responses can be found in my post: "Bring it on, Google!" on the Software Abstractions Blog. (http://blog.softwareabstractions.com/the_software_abstractions/2006/11/bring_it_on_goo.html)

[Side note: Your post on how Mr. Anwar contacted you and got an interview ("How to get ahead") was an inspiration to me as a blogger! I've since contacted many high-level folks in a low-key way for interviews/reactions, and the response has been amazingly supportive!]

[matt coffin]

i love the passion. and i agree. people matter

[Kris Tuttle]

If you can forward chain and backward chain at the same time with a large enough data network, the world is your oyster.